Working with APFS Volume Groups

Printer-Friendly Version
Product: 
ccc5

When Apple introduced the APFS filesystem several years ago, it came with a new concept: the APFS container. All APFS volumes reside within a container, and the container resides within the disk's partitioning scheme. All volumes within a container share the space that is available to the container; separate APFS containers do not share space with each other.

In macOS High Sierra, Apple added the concept of roles to volumes. At the time there were only three roles, and these went largely unnoticed by the average user: Preboot, Recovery, and VM (virtual memory). These roles allow the system to identify specific volumes for specific purposes, and then treat the volumes in specific ways (for example, any volume with the above roles would be hidden by default and also not mounted by default).

The following graphic demonstrates a few of these APFS concepts:

APFS filesystem concepts

The partitioning scheme encompasses the entire physical disk. Within the partitioning scheme you can create one or more APFS containers, and within each container, you can create one or more APFS volumes. Unlike partitioning in the past, all of the volumes within the container share the space that is allocated to the container. In the example above, the three gray helper partitions, the System and Data volumes, and the "Other Volume" all have access to that 700GB chunk of storage. "Other Volume B" is in a separate container, though, and does not share space with the volumes in container "A". Normally a disk would not be partitioned in this manner, but it would be warranted, for example, if you wanted to maintain a clone of your startup disk on that same disk (e.g. for testing purposes by developers).

New concept: APFS Volume Groups

In macOS Catalina, Apple introduced another new concept to the APFS filesystem: volume groups. This is more of a conceptual grouping of volumes within an APFS container, not a new sub-structure. Apple also greatly expanded the number of roles available for APFS volumes (now there are 16 unique roles). When you upgrade to Catalina, your current macOS system volume is renamed, e.g. to "Macintosh HD - Data", its role is set to Data, and then a new volume is added to your startup disk's APFS container with the System role and simultaneously grouped with the Data volume. The two volumes within that group share special bonds and receive special treatment from the Finder and from each volume's filesystem. From the user perspective, these two volumes are treated as a single, unified volume. If you take a look at Disk Utility, however, you'll see the two volumes as distinct, separate items.

The Read-only System volume

Perhaps the single, largest change in macOS Catalina is the manner in which the System volume is mounted on startup – it's read-only. By mounting the volume read-only, it becomes impossible for attackers to make changes to the content of the macOS System volume. That doesn't mean that your Mac is 100% free from all possible attack vectors, rather it's just another line of defense against them.

The Data volume

You can think of the Data volume as a read-write "shadow" of the System volume. The Data volume contains all of your user data (e.g. your home folder, third-party applications), but also contains a handful of system components that can't reside on a read-only volume. For example, Apple has placed Safari on the Data volume, perhaps so it can be updated more frequently. The current startup disk's Data volume is mounted at a special mountpoint on the system. You can find it if you navigate in the Finder to Macintosh HD > System > Volumes > {Data volume name}. What you'll find there is a replica of the System volume's root-level folders. Within these folders are all of the system components that are still writable. Normally you won't see these items in the Finder, though, because the Finder visually mashes the content of the two volumes together to make them appear as a single volume. Also, the Finder won't list your Data volume alongside all of your other volumes – the Data volume is mounted but hidden.

Building bonds with firmlinks

To pull off the illusion of a single, unified volume, Apple added support to APFS for firmlinks. Like the name implies, a firmlink lies conceptually between a soft link and a hard link. That probably doesn't make them any more clear though (even for people familiar with soft and hard links!). A firmlink is described by Apple as a "bi-directional wormhole" between two filesystems. Let's take a look at the "Users" folder as an example – the Users folder at the root level of the System volume is actually a firmlink that points to the Users folder at the root level of the Data volume. If you attempt to navigate to the /Users folder on the System volume, you're actually going to see the content of the /Users folder on the Data volume. Likewise, suppose you're looking at a folder on your Desktop (so you're looking at the contents of the Data volume) and then you navigate upwards several levels. When you get to the parent of the "Users" folder, you're no longer looking at the Data volume, rather that firmlink has transported you back to the root level of the System volume.

There are about a couple dozen firmlinks on macOS Catalina that link various folders on the System volume to writable counterparts on the Data volume. If you're curious about these, you can find a complete list of firmlinks at /usr/share/firmlinks on your startup disk.

Finder shenanigans with the Applications folder

Firmlinks are mostly transparent, but there is one really noticeable exception: the Applications folder. The Applications folder at the root level of the System volume is a firmlink to the Applications folder at the root level of the Data volume, however, if you navigate to your startup disk > System > Volumes > Data > Applications, you'll notice that the bulk of the Applications are not there. Yet when you look at the Applications folder on the System volume, they are all there! The Finder applies some magic here. The read-only System Applications folder actually resides at System > Applications on the System volume, and when you open the Applications folder in the Finder, you'll see the aggregation of that folder and the Data volume's root-level Applications folder. To the average user, this is exactly what you expect to see, and that's great. However, you may notice that this same aggregation is not applied to other system volumes that your Mac is not currently booted from (e.g. your backup disk). On those volumes, if you open the root-level Applications folder on the visible System volume, you'll only see the content of the firmlink to the root-level Applications folder on the Data volume (i.e. no Apple applications, just your third-party applications and Safari). Rest assured, though, that all of the applications are backed up! You'll find them at System > Applications on the backup volume.

CCC will automatically convert your HFS+ formatted destination to APFS to support backing up a Catalina+ volume

Because macOS Catalina leverages volume groups for the startup volume, creating a bootable backup requires an APFS formatted destination volume. HFS+ is no longer an option for booting macOS starting with macOS Catalina. For your convenience, CCC will automatically convert your HFS+ formatted backup volume to APFS as necessary and create a volume group on the destination. This conversion is the same conversion that took place on your startup disk when you upgraded to High Sierra or Mojave, with one notable exception: CCC tells you that it's going to convert the destination, and gives you the opportunity to decline the conversion. The conversion is non-destructive — any data that you have on the destination volume will remain in place, the only thing that changes is the format of the volume.

Why might I not want to allow the conversion of my destination volume?

Typically there is no reason to decline the conversion. The conversion is non-destructive, and it's required for making a backup of the system. If your backup volume is dedicated to your CCC backup task, then converting the destination to APFS is the right choice.

However, if your destination volume is not dedicated to your CCC backup task or if you're not intending to back up the macOS System files, you should consider how the other uses of your destination might be affected by the conversion. For example, Time Machine is not currently compatible with APFS as a destination, so converting a destination volume that contains a Time Machine backup would break the Time Machine backup. CCC specifically avoids converting Time Machine backup volumes. Another example – if you're only backing up a single folder or handful of folders from your startup disk, you should configure a folder-to-folder backup instead, which won't require any conversion of the destination.

Alternative to conversion: Create a dedicated partition

If you're using the destination volume for non-CCC work, and you cannot or prefer to not convert the volume to APFS, you can create a dedicated partition on your destination disk for CCC to use. To create the partition:

  1. Open Disk Utility
  2. Select your destination disk in Disk Utility's sidebar
  3. Click the Partition button in the toolbar
  4. Click the "+" button to add a partition to the disk
  5. Set the name and size of the partition to your preference
  6. Choose APFS as the format
  7. Click the Apply button

How long will conversion take?

It depends on how much data you have on your destination volume, the performance of the destination device, and the degree to which the destination volume is fragmented. It can take a while, but CCC won't wait for more than two hours for the conversion to complete. If it's taking longer than two hours, then CCC will recommend that you erase the destination volume instead, which will resolve any performance issues that are directly caused by filesystem fragmentation. If CCC issues this recommendation and you prefer to wait out the conversion rather than erase the volume, you're welcome to convert the volume in Disk Utility instead (the option is in the Edit Menu).

Encrypted volumes and APFS volume groups

Encrypted HFS+ volumes cannot be automatically converted to APFS, and CCC cannot create an APFS volume group using an encrypted APFS volume. When you select a Catalina+ startup disk as a source and an encrypted volume as a destination, CCC will disallow the selection and suggest that you erase or decrypt the destination volume. Erasing the destination volume is the simplest approach, and you can find detailed instructions for doing that here: Preparing a hard drive for use with Carbon Copy Cloner

If you're making a backup to an encrypted volume that you've backed up to before (e.g. on Mojave and previous OSes), you may not want to erase all of the data on that volume. You can decrypt the destination volume with one of the following methods:

A: Boot from the backup volume, open the Security Preference Pane, disable FileVault

B: Decrypt the volume in the Terminal application. E.g. for an HFS+ formatted destination:
diskutil cs decryptVolume "/Volumes/CCC Backup"

Or for an APFS-formatted destination, get a list of user IDs associated with the encrypted volume, then use one of the "Local Open Directory User" UUIDs from the output of the first command with the second command:
diskutil ap listUsers "/Volumes/CCC Backup"
diskutil ap decryptVolume "/Volumes/CCC Backup" -user B44348A3-68DF-4B7B-800D-47FE38711178

Replace "B44348A3-68DF-4B7B-800D-47FE38711178 " with a UUID produced by the first command.

Wait for decryption to complete

If you've chosen to decrypt the destination rather than erase it, you'll have to wait for the decryption process to complete before you proceed with your backup task. Decryption will continue in the background while you're booted from your production startup disk. macOS doesn't offer a convenient method to see conversion progress, but you can type diskutil apfs list (or diskutil cs list if the applicable volume is HFS+ formatted) in the Terminal application to see conversion progress.

Re-enabling FileVault on your Catalina backup volume

After you have run your backup task to a non-encrypted volume, you can then boot from the backup and re-enable FileVault in the Security & Privacy Preference Pane.

If I decrypt or erase the destination, then reenable it later, will I have to do this again for future backups?

No, this is a one-time task that is required for CCC to be able to make adjustments to the destination volume that are required for macOS Catalina. Once you have established a Catalina backup, you can reenable FileVault and your future backups will work without any additional intervention.

Related Documentation

Tags: