Blog

We field a lot of support requests, and similar to a doctor's office, we see some extreme cases. One of those interesting extreme cases are folders with high file counts. Any time a folder has more than a few thousand items in it, the filesystem is going to be a lot slower when working with that folder. Adding a new file, for example, requires that the filesystem compare the new item name to the name of every other file in the folder to check for conflicts, so trivial tasks like that will take progressively longer as the file count increases. Gathering the enormous file list will also take progressively longer as the list gets larger. The performance hit is even more noticeable on rotational disks and network volumes, so we often see these sticking out in backup tasks.

Sometimes high folder counts can bring a backup task to a halt

Last week, one of our users found the task as shown above. Upon closer analysis, we determined that the "media" folder had 181,274 files in it. In other words, more than 10% of the files on the whole startup disk were in that "media" folder. In extreme cases like this, the delay to retrieve a file list can be so long (i.e. longer than 10 minutes) that the task aborts with an error, e.g.:

The task was aborted because a subtask did not complete in a reasonable amount of time. Or,

The task was aborted because the destination filesystem is not responding.

These are typically pretty wild cases that benefit from some human intervention. For example, I recall seeing an AddressBook application support folder that had more than 2.5 million image thumbnails in it. Nobody has that many contacts! That wasn't an isolated incident, either, we've seen that same AddressBook folder implicated at least a half dozen times. Cases like that usually point out a software failure in the application that is creating the files – a failure to sanity-check its data store, or to put a sane limit on how many log files it creates. Sometimes, like in the Telegram example above, it's just short-sighted design.

For a contrasting example, consider how Mail organizes a potentially astronomic list of files. If you navigate to the hidden Library folder in your home folder, then to Mail > V10 > {any UUID} > {any mailbox} > {another UUID} > Data, you'll see folders named by number, four layers deep, until you finally get to a Messages folder with actual files in it. While this nested hierarchy makes it annoying for a human to find a specific email file, that organization limits any individual folder's file count and optimizes Mail's ability to quickly collect those resources from the filesystem.

Some data stores with potentially-high file counts

Here's a list of some common culprits we'... Read More

We've had so many new features in the oven for a while, and now we're finally ready to share it with the world! CCC 6 offers unprecedented accountability for your backups and insight into what's changing on your Mac, plus a brand new file copier that's faster, smarter, and designed to adapt to Apple's fast pace of OS and filesystem innovation.

Take a look at what's new in CCC 6:

• Faster backups with our next-generation file copier
• Quick Update: Update your backups up to 20x faster
• Redesigned interface with Dark Mode, real time task performance info and estimated time remaining
• Snapshot navigator: Easy way to explore older versions of files
• CCC Dashboard: The new menubar app, now with snapshot disk usage
• Run backups "When files are modified on the source"
• Task Preview: See what will happen before you back up
• Backup Audit: Review what was copied and why
• Compare: Visual comparison of the source and destination
• Advanced file verification options
• Pause a backup, and several other features our users have asked for

The core CCC backup features you know and love are now better than ever!

Faster backups with our next-generation file copier

We've completely rebuilt our file copier to take advantage of the performance characteristics of modern storage. Built on a multi-threaded design, our new file copier provides a foundation for many of the new features noted below, and paves the way for many new features in the future.

Quick Update: Update your backups up to 20X faster

CCC can now tap into the macOS FSEvents service for a list of folders modified on the source since the last backup rather than scanning every folder for changes. Especially for tasks involving a destination network volume, the performance benefit of this feature cannot be overstated!

Redesigned Interface with Dark Mode

It’s been a while since we’ve given CCC’s interface an update and think you’ll enjoy its sleek new look. CCC’s main window is reorganized to make it smaller while making many of the controls and font sizes larger. We completely redesigned every element in CCC to offer a high quality Dark Mode experience. CCC now offers more detailed progress indication while a task is running, including a time remaining estimate. File processing and transfer rates are now... Read More