Shirt Pocket Discussions  
    Home netTunes launchTunes SuperDuper! Buy Now Support Discussions About Shirt Pocket    

Go Back   Shirt Pocket Discussions > SuperDuper! > General

Reply
 
Thread Tools Rate Thread Display Modes
  #1  
Old 05-12-2005, 01:43 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Data integrity question- how much verification by SuperDuper?

What kinds of verification tests does SuperDuper perform to ensure data integrity of the backup copy?
________
VAPOR TOWER

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #2  
Old 05-12-2005, 05:02 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)

Yes, the built-in verification prevents media errors or device errors... but, it does not prevent errors or verify that the files are copied correctly to begin with. (For example, are all files copied? are permissions correct? etc.)

Also, the SuperDuper documentation specifies that "[files which Apple recommends not to copy]" are not copied... yet, there is no explicit documentation or verification to determine exactly what is copied and what is not copied.

Thus, it is not possible to determine if the backup copy is identical to the original or not. Presumably, it is not identical... but, not knowing the exact differences, makes it more difficult to determine if data integrity was maintained or not.
________
Detroit diesel v8 engine

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #3  
Old 05-12-2005, 05:04 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?

The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.
__________________
--Dave Nanian
Reply With Quote
  #4  
Old 05-12-2005, 05:17 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?
I don't wish to argue with you. But, availability of a verification feature would at least indicate some concern for data integrity on your part.

On a technical level, it seems that verification is less prone to troubles that plague copying and backup utilities on OS X.

There are lots of other reasons why verification is valuable and important, which I am sure you don't need me to list here.


Quote:
Originally Posted by dnanian
The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.
Thanks, that's helpful!
________
Ford falcon (argentina) specifications

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #5  
Old 05-12-2005, 05:28 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
I'm not trying to argue: I just want to make sure you understand our position.

We spend a lot of time testing and ensuring that we do the right thing when we copy. We obviously have a high regard for data integrity, and spend considerable effort ensuring that we do the right thing, and the software is never released to make a date: it's released when it's been thoroughly tested.

We have internal tools that do careful comparisons of source and destination to ensure we're doing the right thing, and we don't release versions that don't pass our extensive battery of tests.

Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.

If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.
__________________
--Dave Nanian
Reply With Quote
  #6  
Old 05-12-2005, 05:36 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.
Thanks. I understand your position, but very strongly disagree with your opinion and logic here.

Obviously, you rely and trust your verification tools and consider them very valuable to you. So, it makes little sense for you to turn around and say they would have little or no value to anyone else.

Do you think you are the only person concerned with matters of data integrity?

How do you think other people would want to trust your software, if you don't even want to make available tests to verify the integrity of its copying?




Quote:
Originally Posted by dnanian
If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.
Of course.

In part, my questions are related to resolving differences in results of using various other tools, both to perform copying tasks and verification tasks.
________
Fz6

Last edited by stevea; 03-10-2011 at 11:47 AM.
Reply With Quote
  #7  
Old 05-12-2005, 06:37 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure. Here's the original text.)

We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)

Hope that helps!
__________________
--Dave Nanian
Reply With Quote
  #8  
Old 05-12-2005, 07:02 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure.
I first posted the original message in a different thread, then, after I realized my mistake, deleted it and reposted here as a new thread.

You were just too quick, and replied to the first message that I deleted in the other thread!
________
Ford nucleon picture

Last edited by stevea; 03-10-2011 at 11:48 AM.
Reply With Quote
  #9  
Old 05-12-2005, 07:05 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Oh, Jeez, good. I was becoming concerned that things were well and truly hosed here!
__________________
--Dave Nanian
Reply With Quote
  #10  
Old 05-16-2005, 12:19 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Just to illustrate, here is an important scenario which is not addressed by SuperDuper, but where verification is essential to maintain data integrity:

Copy Disk 1 to Disk 2 using SuperDuper

During the copy, contents of Disk 1 (or Disk 2) are altered. This can happen very easily, because both disks are available to the user during the copy operation. It's possible for the user to make changes, or other applications to make changes, with or without the user being aware of it.

Without verification, the user believes Disk 1 and Disk 2 are identical.

However, in this situation, they are not.

Verification not only reveals the existence of the differences in data on Disk 1 and Disk 2, but also pinpoints and identifies the exact differences, so the user can be aware and determine how to proceed.
________
HOW TO ROLL A BLUNT

Last edited by stevea; 03-10-2011 at 11:49 AM.
Reply With Quote
  #11  
Old 05-16-2005, 12:39 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Thanks for the example, Steve.
__________________
--Dave Nanian
Reply With Quote
  #12  
Old 05-22-2005, 11:42 AM
marianco marianco is offline
Registered User
 
Join Date: May 2005
Posts: 1
Steve,
I don't think your example is a good one. If a user deliberately makes changes to a disk while SuperDuper is running a backup, they should automatically realize that the changes may not be copied over the the backup. The solution would be to run SuperDuper again to update the changes. If anything, the user should refrain from using the Mac while running SuperDuper to avoid problems like this.

I also thought Dave's answer was pertinent and sufficient. The disk driver itself does the verification of the copy, otherwise verification becomes impractically time-consuming. SuperDuper's cloned disks have been the best, cleanest, most problem-free I have obtained from any utility so far. This includes Synchronize Pro (which I now use only for synchronization of folders) and Carbon Copy Cloner. What makes it so good is that even the aliases point to the correct files in the clone. Note that Synchronize Pro does have a simple veriication procedure (checking file size, various flags, etc) which is relatively quick. I have used this, but it hasn't been very useful since too many files are listed as different, when the data in the file I know is identical. After a while, I ended up ignore the listing of files the verification process made in Synchronize Pro's log file.

Your initial scenario of copying a drive with known severe directory/corruption problems is also problematic. I don't think you can reliably get a clone of that drive in the first place, since the files themselves cannot be reliably found. By the way, I did have such a drive once - where Disk Utility and Disk Warrior both reported the drive was too damaged to use. In this case, I don't know which helped but I ran Norton Utilities' Disk Doctor (which is why I still keep it around) and did a Safe Boot (holding the shift key while starting up, which runs the fsck command-line utility), and the drive was repaired. If the drive is corrupted beyond the usual repair utilities, you may have to use Data Rescue (a third party utility), to scavenge the drive and attempt to recreate its files and structure on another drive. Data Rescue is probably the best at doing this.

If you really want to verify the exact file differences between the original and the clones, a utility that can do this very well is "You Synchronize". It uses CRC32-bit checksums to determine if files are different before synchronization. This is far more sophisticated than checking file lengths or modification dates. This will definitely tell you what differences there are between the original and clone. The problem is that if you have hundreds of thousands of files (as many people often have these days), it literally takes hours upon hours to complete the verification since You Synchronize has to do the CRC 32-bit checksum for each file on the original and each file on the clone, then compare the checksums then add the data to its database. For myself, this made it very impractical to use on a daily basis - particularly since I like backing up to four clone hard-drives daily. I would be limited to one overnight synchronization a day if I used You Synchronize on anything but a small number of files.
------
marianco
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
PROBLEM: (BUG?) Copied disc (or not?)- files not visible in Finder stevea General 7 05-13-2005 10:47 PM
Minor concerns regarding SuperDuper giba General 1 05-02-2005 05:06 PM
Security of SuperDuper Backup Data? Zeigh General 1 08-08-2004 12:29 PM
Another review: MaMUGs looks at SuperDuper! dnanian General 0 01-26-2004 09:26 AM


All times are GMT -4. The time now is 10:25 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2024, vBulletin Solutions, Inc.