Image grows without control

[jgrimaldos]

Hi, I'm getting a weird issue when using SuperDuper.

The backup process starts ok, but when the backup is around 75-80%, the progress bar stops growing, and all the counters stop yet the image keeps growing and growing and growing until i run out of space... mind you, the original disk is around 100GB and the image grows up to 200GB!

Any ideas about this issue?

[dnanian]

It's likely that a large file, or a constantly updated one, is being copied at that point. Do you get an error that supplies a filename?

[jgrimaldos]

No, i don't get any errors apart from the size one.

[dnanian]

OK. Please send me the log using the "Send to shirt pocket" button and I'll take a look.

[Drdul]

I'm encountering the same problem. My image files (one created daily, one created weekly with "smart update") are around 250 GB, when they should be about 60 GB. When I mount one of them and have a look, I can only find about 61 GB of stuff — why the almost 200 GB of extra file size?

[Drdul]

FWIW, the problem isn't related to "smart update." I just recreated an image using the "erase and copy" option. I watched as SD progressed. It reported copying 62.4 GB of files, yet the resulting image occupies 246.6 GB of space!

[dnanian]

Erase, then copy does not recreate the image: it erases the contents of the image and then re-copies.

The deal here is pretty simple, as I've tried to explain before: Apple preferentially grows the image to use "new" space rather than re-use free space in the image. No doubt part of that is to preserve "erased" information as long as possible.

Anyway, you can shrink an image with "hdiutil" as shown elsewhere in the forums -- that's the best thing to do. Or, rather than doing an "erase, then copy", just delete the image. We'll recreate it.

[Drdul]

Okay, I'll do that, but unfortunately it doesn't prevent the problem from occurring again. I'm using a 1 TB hard drive for my images, and because they're growing so much I only have 100 GB of space left. If they get any bigger, I won't be able to use Super Duper any more — hardly a good situation.

Given that hdiutil is available, how about adding it as an option within Super Duper? It could run "on successful completion," for example. I know that I could probably write some kind of AppleScript thingy to run hdiutil once a day, but I'm not an AppleScript expert — it would be nice if SD could take care of this for me.

Cheers!

[dnanian]

If you want to set a different maximum size, you can create the image with Disk Utility, then point SD! at that image. It will not grow to larger than the size you set.

[Drdul]

Hi Dave:

Thanks for the suggestion. How exactly do I create a, say, 90 GB image with Disk Utility (I know how to create disk images of selected files, but I'm not clear what i need to do to create a disk image for SD to use).

In any event, this suggestion highlights an important issue. One of the selling features of SD is its ease of use. My mother could use SD! Yet in order to avoid a fairly major problem of disk images expanding out of control, a user has to be pretty knowledgeable — we're certainly well beyond "easy" at this point, and there's no way my mother could deal with this. Is it not possible to add something to SD to automatically shrink disk images back down in size, with hdiutil or something similar?

Cheers!

[dnanian]

It's important to recognize that the images are not growing "out of control". They're growing up to their maximum size, which is the size of the source volume -- which could happen regardless, because that's the size of the files you could have. The image file does not "hug" its contents.

I understand the concern here, but for most people, the idea is to have enough space on the destination to fit the files on the source... constantly compacting the images takes a huge amount of time and does a *lot* of additional I/O, which isn't generally a good idea...

Anyway, to create the image, This is a known problem in SD! with 10.3.x that we'll be fixing in the next release. In the meantime, the workaround is to create the sparse image, of the appropriate size, using Disk Utility. You'll useonly have to do this once.

1. Open Disk Utility (found in /Applications/Utilities).
2. Without selecting any volume/drive on the left side, click the New Image toolbar button.
3. Give the image an appropriate name (like Backup), and store it on a drive other than the one you're trying to back up.
4. In the Size pop-up, choose Custom.
5. In the sheet that appears, enter the maximum size for the image. Note that under Panther, the pop-up does not have a selection for GB: you need to enter "40000" MB for 40GB. Click OK when you've got the right size entered.
6. Decide whether or not you want the image encrypted, and choose the proper value in the Encryption pop-up.
7. In the Format pop-up, choose sparse disk image.
8. Click Create.

Once it's created, go back to SuperDuper! and choose the "Disk Image..." selection in the destination pop-up. Navigate to the location where you created the image and click it (it's grey, but that's OK -- click it anyway), then click "Save".

That should do it!

[Drdul]

Thanks for the instructions. Does this mean that the next version of SD will include a feature to specify a maximum size of an image file when creating it? If so, that would address the issue nicely, so that my Mom can still use it.

[dnanian]

That's not generally the plan, no. Since your source volume has a given size, it only makes sense to have the destination reflect that same size. Your personal requirements might be different, but for most people the approach we've taken is the correct one.

(In fact, for most people, it's best to back up directly to the drive, rather than to an image.)

[Drdul]

I get what you're saying, but here's the thing. SD includes the ability to exclude specific directories from an image. My source drive is 750 GB, and is about half full. I have exclude my Documents, Movies, Pictures and Sites folders (as these are backed up separately with a different app), leaving just the "system files" — basically, what I would need to clone a new hard drive if the one I'm using now suddenly melted down (or my computer was stolen). That's why I only have approx. 60 GB of files.

Because I'm using a 750 GB hard drive, does that mean my SD backup could grow to 750 GB, even though I'm only backing up 60 GB? If that's the case, it would be really nice to have something built into SD to limit the size of the disk image, as my Mom wouldn't be able to do it using Disk Utility.

With the impending release of Leopard and the built-in Time Machine feature, a lot more people might find themselves wanting to back up just the system files, as Time Machine would be taking care of their "user files." This thing with the image file being several times larger than the files it contains might be a more widespread issue. In anticipation of that, you might want to add a "system files only" backup script to SD and a built-in ability to limit the size of the image file.

In the meantime, I've got my solution (using Disk Utility), so I'm happy. Thanks again for taking the time to explain all this.

Cheers!

[dnanian]

Understood. But I think you'd find that, even with Time Machine, a full, bootable backup with all your files -- kept up to date -- is going to be quite useful. I don't want to encourage people to make "system file only" backups, because -- believe me -- your Mom is not going to be able to restore and reintegrate data from multiple sources. But she will be able to start up from a full copy on an external drive and restore...

(See my Time's Arrow post about SD! and Time Machine for more.)