Silencing a cloud storage echo chamber

A routine backup that should have finished in minutes was instead dragging on for hours. The culprit turned out to be a tiny menu bar clock app that had quietly sprouted a million empty folders. Here's how we tracked it down – and how one very persistent customer finally dug his way out.

The user, "John", noticed his backup task was taking much, much longer than expected. He had CCC submit logs alongside his request, so I was able to review the details of the events. Two stats in the prescan result stuck out prominently:

📊 Totals: files: 2,622,988, symlinks: 3,766, folders: 1,083,778, data: 601.9 GB. Scanned in 15353s

4 hours to enumerate the source! I checked the source device details – it was the startup disk in an Apple Silicon Mac, so definitely not slow. Other context indicated that the startup disk performance was generally just fine.

The smoking gun

The folder count was suspicious, too – about 5x higher than usual for that file count. After slogging on a while longer, CCC's stall detection flagged exactly where the task was getting stuck:

🔺 Stall encountered at /Volumes/Backup of Macintosh HD/Users/john/Library/CloudStorage/Dropbox/John User/MenuClock.app/Contents/Resources/Base.lproj|:|0|:|1060148

The two numbers at the end of that path are the file and folder count for this folder. That "Base.lproj" folder has no files in it, but 1 million subfolders. That's absolutely bonkers! That really can't be sane. The app, MenuClock (not the real name), looks like it's just a simple digital clock, so it really shouldn't have many items in that folder at all. 

I offered my best guess: I suspected that some conflict arose in that application's bundle specific to Dropbox, and Dropbox ran amok, creating lots of folders (perhaps remotely, then locally, like an echo chamber). 

I recommended removing that app from Dropbox on the startup disk, then removing it from the backup volume too:

  • Open "Macintosh HD" in the Finder and navigate to Users > john
  • Press Command+Shift+Period to toggle the Finder's display of hidden items
  • Continue to navigate to Library > CloudStorage > Dropbox > John User
  • Toss that MenuClock.app item in the Trash
  • Repeat for the "Backup of Macintosh HD" volume – navigate to Users > john > Library > CloudStorage > Dropbox > John User and toss MenuClock.app in the Trash
  • Empty the Trash [...] and leave your Mac for the weekend – it will probably take a while for the Finder to slog through 1M folder deletions (but Finder will be more patient than CCC, so that's my recommendation for this one-off removal).

Followup

I anticipated that it would take quite a while to slog through the removal of that content. John noted, 

Thanks for pointing out the problem to me. I’m trying to delete it but it hangs the Mac when I try to delete the directory. I was able to delete it from Dropbox itself, but one machine is so messed up (my main iMac of course) that I really can’t use it and am considering a wipe… even after a reboot it’s not happy.

Two weeks later John followed up with a positive conclusion:

I had 3 iMacs, 2 iPhones, 4 MacBooks and the Dropbox Cloud to clean up (whew!)

The first deletion was the hardest as the system was bogged down by cleaning up the 1,000,000 files (of unknown size) for each machine. At least Dropbox deleted immediately, although I had to repeat it a few times as the connected Mac machines put it back up there.

After I let the deletes run for a day or so each I found that the suspect file kept reappearing on different machines at various times. The problem was that they were all connected via the Dropbox Cloud. I kept deleting them as they appeared, but I could never seem to catch up. Finally I put a Finder window for each machine’s problem folder and the host spread between my two side-by-side iMacs. I then had a ‘control panel’ where I could see the changes happen and stomp on them. It still took awhile, but after a day or so I am problem free, and my CCC backups are running to completion. I hate to think of these files on one Time Machine backup.

We never did determine the exact underlying cause for the propagation of folders in that application bundle, but the reappearance of the item on various devices after removing it elsewhere suggested that the cloud-syncing software was likely (errantly) recreating the application and its subfolders. With a lot of persistence, John was finally able to eradicate them.

What to watch for

Luckily, most people will never have a million-folder app bundle, but the underlying lesson applies to anyone who is including cloud files in their backups. Cloud storage providers (like Dropbox, iCloud Drive, OneDrive, Google Drive) attempt to resolve inconsistencies in the background, so when a sync conflict goes sideways you can get unexpected results.

If a task that used to take minutes is suddenly running for hours, take a look at CCC's task history. In the errors tab, you might see a stall like this listed:

The filesystem did not deliver the content list for this folder within the time limit specified in CCC Settings > Advanced. You can set a longer timeout to work around this problem, but there may be a sanity problem (e.g. too many items) or some other filesystem anomaly that is causing the problem.

This is CCC's stall detection at work. You can read more about what triggers it – and how to adjust the stall timeout – in the CCC Knowledge Base.

We're here to help

If something doesn't seem right in your backups, choose "Ask a question" from CCC's Help menu – we're always happy to review your backup task events and configuration and see if something can be changed to improve the result.