Expectations on copy speed

[AnalyticsUSA]

While I understand that there are many variables which effect copy speed, knowing what is normal, or conversely not normal, can help to identify when there's a system vs hardware vs software issue.

I've seen various people posting firewire "effective copy speeds" which are quite impressive, however, so far the best my configuration has been able to produce is 2.08 MB/s. Seems a bit sub standard...

The configuration is an Intel Mac @2Ghz, OS X 10.4.8; 2GB ram; connecting to a 500GB LaCie Ethernet NAS with EXT3 over 100MBps local net. I know that a firewire might be more effective, however it sort of defeats the purpose of an NAS. Besides, I don't see any numbers in this configuraiton which would drop the throughput to 2MB/s.

Thanks in advance

[dnanian]

That sounds about right to me, actually. There's a lot of general inefficiency when growing an initial sparse image on a NAS. (The fastest I've seen is the Infrant ReadyNAS running on Gig-E, though.)

[xochi]

My general experience on reasonably fast machines (doing backups to local internal or external firewire drives) is that I get average actual copy speeds in the range of 5-15MB/sec for a fresh backup (Erase, then copy), and can get effective speeds in the 50-150MB/sec range for Smart Updates when less than 10% of the drive needs to be copied.

It's clear that the Erase then Copy speed is slow -- it is clearly not disk-speed bound (since FW400 can do 40MB/sec). I believe that these speeds could be a lot faster if the comparison code were optimized. Whether this slowness is in Apple's OS or in SuperDuper's code is not clear...

[dnanian]

This really isn't true, xochi: we do no comparisons when doing an erase, then copy, because there's nothing to compare to. The slowness is the file system dealing with hundreds of thousands of small files, their metadata, etc.

[xochi]

If the bottleneck is in writing tons of small files and the problem is really due to drive head seek speed, is there any way to speed it up? Can you do the copies using a mode that says "it's ok to cache small file writes on the destination drive?" I know in some cases you can tell the OS whether you must have the file written to disk now, or if it's ok to cache it for a bit...? Not sure if this would help at all.

[dnanian]

The caching is always on, xochi, and the files are not explicitly flushed.

[xochi]

I guess this answers the question "why is it better to have fewer large files (like OS 9 used to do) instead of a larger number of small files (like Unix does)?"

[dnanian]

Nah, not really, just shows some of the problems with HFS+. But once copied, evaluation happens quickly, even with the zillion files...