Skeletonized, Near Empty Backups Silently Created

[Guy Kuo]

I've been a regular user of SuperDuper for some time. It is a tool I regularly use to efficiently create "smart copy" clones on external drives. These I do on a rotating basis so my backups are grandfathered. Tonight I had a shocking discovery post a severe drive failure. When my MacBook drive became corrupted and was irreparable, I wasn't bothered much. I knew I had some recent SuperDuper backups from which to restore my drive.

I was wrong.

My most recent two backups (create under OSX 10.5.2) were both nearly empty despite the smart copy process which created completing successfully. I was shocked to find that my cloned drives were populated by folders but practically no files. It was as if SD had recreated the folder structure, but didn't actually copy any files into the folders. Shocked does not describe the sinking sensation of knowing my last two backup drivess were essentially empty skeletons of only folders.

How this could happen I do not know, but it surely does shake my confidence in SD. I've recommended the program and used it religiously, but it should have NEVER appeared to create a clone when it didn't actually do so. Those three green bars at completion time no longer mean for me that all the files were copied. Apparently, one must now check the clone after it is smart copied to see if anything is actually still on the drive.

Thankfully, I also back up (albeit less frequently that with the SuperDuper routine) by other means. So, my data loss isn't complete, but this really should never be able to happen without warning.

[dnanian]

I have so little to go on here, Guy, that I don't know how to respond. I don't know of any cases where a copy would indicate success and then copy "nothing", nor anything where files would be mysteriously "skipped", if you're using the standard "Backup - all files" script.

What folders were empty? What script were you using? Do you think that, perhaps, your drive was corrupt enough when you made the backup that the system wasn't returning the right files in the first place?

[Guy Kuo]

Originally Posted by dnanian

I have so little to go on here, Guy, that I don't know how to respond. I don't know of any cases where a copy would indicate success and then copy "nothing", nor anything where files would be mysteriously "skipped", if you're using the standard "Backup - all files" script.

What folders were empty? What script were you using? Do you think that, perhaps, your drive was corrupt enough when you made the backup that the system wasn't returning the right files in the first place?

Unfortunately, because the drive from which the SD was run is no longer existent, I can't provide a copy of SD's preferences or settings. I've never even tried to do a script with SuperDuper. I have just smart copied. Until now, the program has always done the job. I really missed SD when we were awaiting Leopard compatibility.

I guess the lesson here is to ALWAYS inspect the clone drive and verify that files were actually stored even if SD seems to report all went well.

As for what folders were copied, at the root level I see only...

Users
Library
Applications <-- an empty folder
System

And if I open System I see only....

Library

Open that and I see only...

Caches

Open that and see...

com.apple.bootstamps <-- which is an empty folder


Looking inside the root Library folder, I see...

Preferences
Caches

Inside Preferences one sees...

FLEXnet Publisher <-- an empty folder

Inside Caches I see...

REALbasic <--- an empty folder


So, whatever caused SD to prune files (and folders) from the clone, did so very dramatically. What is most upsetting is that the copying process seemed to proceed normally. The time needed wasn't noticeably different from my normal smart updates.

Here I am cobbling back together over 100 GB of data from multiple sources instead of expectedly recovering from a recent SD generated clone.

Heck, I'm using SD to bring back an older clone as part of my recovery, but this has been a real shocker.

[dnanian]

I really have no idea what happened here, Guy. It really sounds like the source drive was corrupt during the copy, which also caused the backup to be corrupt. This is, at some level, confirmed by the fact that you had post-backup corruption.

SD!, of course, can only return errors if it knows something is wrong. If the OS is providing us with an incorrect list of files, we're going to step through them normally, and do what's necessary to make the destination exactly correspond to the source.

In this case, at least as far as I can piece together the narrative, it looks like it returned rather incorrect data... without any indication of an error.

Did you run this backup manually, or was it on schedule? Running anything low-level like an AntiVirus program? Any recent weird failures on your system that might require a hard power down, unresponsive applications, things like that?

[Guy Kuo]

Originally Posted by dnanian

I really have no idea what happened here, Guy. It really sounds like the source drive was corrupt during the copy, which also caused the backup to be corrupt. This is, at some level, confirmed by the fact that you had post-backup corruption.

SD!, of course, can only return errors if it knows something is wrong. If the OS is providing us with an incorrect list of files, we're going to step through them normally, and do what's necessary to make the destination exactly correspond to the source.

In this case, at least as far as I can piece together the narrative, it looks like it returned rather incorrect data... without any indication of an error.

Did you run this backup manually, or was it on schedule? Running anything low-level like an AntiVirus program? Any recent weird failures on your system that might require a hard power down, unresponsive applications, things like that?

The backup was run manually. I never schedule.

One easy feature addition could keep this from happening silently. If SuperDuper gave a notification when it detects that the destination volume at the end of a clone operation has considerably fewer than when it started. i.e., drops more than 15% in file count. This could be made an optional notification and would not require SD to do much more calculation.
Such a simple warning would have prevented my continuing to produce multiple successive near empty backups. Every time that happened silently, I was unknowingly getter further and further away from a good backup. Since, SD's main purpose is to ensure data integrity across disasters, such a warning would be very welcome and need not be intrusive.

Not complaining or blaming, but I'd love to see this become nearly impossible to happen without noticing. Even if there is an errant structure on the source drive. I'd sure like to know if the clone drops in count from what the file count it started at.

[dnanian]

We do display the total of the files (etc) in the main window, Guy -- it would display a pretty low number if it was skipping nearly everything as you've indicated here (and, again, we've never seen before in all the years SD! has been out).

It's really hard for us to display a "warning", because a script can do pretty much anything, and even a regular "Backup - all files" could be skipping tens of thousands of temporary files, depending on how long your Mac has been up for, and what you've been doing.

Did you have more than one backup?

[Guy Kuo]

Originally Posted by dnanian

We do display the total of the files (etc) in the main window, Guy -- it would display a pretty low number if it was skipping nearly everything as you've indicated here (and, again, we've never seen before in all the years SD! has been out).

It's really hard for us to display a "warning", because a script can do pretty much anything, and even a regular "Backup - all files" could be skipping tens of thousands of temporary files, depending on how long your Mac has been up for, and what you've been doing.

Did you have more than one backup?

I still think it would be good to have an optional warning dialog. I don't always read the fine print in the program's output, but I do see three green bars indicating my backup was successful. I know that the little text inside the bars can could indicate the number is off, but heuristically, seeing three green bars indicates that the backup process was successful. It would be worth adding that extra safety measure for SD users. You are already calculating the numbers and it would require virtually no effort to add a dialog box which appears and notifies users if a target drive was dropped by a large percentage.

Yes, this may be intentionally done by some scripts and you cannot predict the script action, BUT it would be trivial to implement and very useful a warning for those of us who use the program to perform backups. At the very least, throw up a warning if the script is a "backup all", but the target drive ends up with a smaller number of files than the source.

Take this as a very strong feature request. Add an option to display an INFORMATIONAL warning if the target file count is dropped by a significant percentage. Even if I were running a script that intentionally causes such a drop, it wouldn't harm me to have a notification. However, if this happens and a user didn't intend it to occur, it would be infinitely better to have this plainly pointed out before further data loss is propagated.

Thankfully, I had some other Finder created backups from which to pull files and rebuild. My two most recent SD backups were a total loss.

[dnanian]

Nothing is ever trivial to implement, Guy... but I do understand your request. Thanks.

[Guy Kuo]

Thanks for considering it. I love the speed of SD's smart copy. That is the number one reason I use it.

I'd just like to reduce the chances of anyone ever going through what I did. It's a very bad feeling when more than one generation of backup turns out to be bad. Even worse when you had no indication and would have otherwise continued to create further generations of bad backups. Every cycle would deepen the data loss. Eeek.

[dnanian]

Please do recognize that this has nothing to do with Smart Update. A full copy would likely have done the same thing, given the damage that seems to have been present.

[Guy Kuo]

Originally Posted by dnanian

Please do recognize that this has nothing to do with Smart Update. A full copy would likely have done the same thing, given the damage that seems to have been present.

Sorry, I didn't mean to imply that Smart Update was the culprit. I was pointing out a very positive feature of SD.