How would I speed up a full disk dd?
In my experience, I don't think there is something faster in the command line as dd
. Adjusting the bs
parameter can increase the speed, for example, I have 2 HDD that I know have a read/write speed greater than 100 MB/s so I do this:
dd if=/dev/sda of=/dev/sdb bs=100M
There is also pv
(Needs to be installed first) that checks for the fastest speed on both drives and then proceeds on cloning. This has to be done of course from root:
pv < /dev/sda > /dev/sdb
With PV I got 156 MB/s
The nice thing about pv
apart from the speed is that it shows the progress, current speed, time since it began and ETA. In regards to HFS+ I would not know, am just trying to help on the "speed" part. With pv
or a very optimized bs
parameter, you can do a 4 TB drive in less than 7 Hours (6 Hours 50 Minutes at a current speed of 150 MB/s).
I did a couple of tests with the connection types you were using and others I had available. I was using the Asus Z87 Pro and the Intel DZ68DP. This were my results, but first we need to know that the theoretical speeds for many transfer rates (Raw speeds) are just that, theory. Doing real tests revealed they are between 40% to 80% of that raw speed. This tests can change depending on Device used, connection type, motherboard, type of connecting cable, filesystem type and more. With that in mind, this is what I got (I only tested Write speed to the Device, read is typically higher):
Connected Device - Connection Type - Speed (Write Speed)
USB 2.0 USB 2.0 25 MB/s
USB 3.0 USB 2.0 35 MB/s
USB 3.0 USB 3.0 73 MB/s
eSata eSata 80 MB/s
Sata 2G HDD Sata 2G 120 MB/s
Sata 3G HDD Sata 2G 140 MB/s
Sata 3G HDD Sata 3G 190 MB/s
Sata 2G SDD Sata 2G 170 MB/s
Sata 3G SDD Sata 2G 210 MB/s
Sata 3G SDD Sata 3G 550 MB/s
To copy a partition wholesale, use cat
instead of dd
. I ran benchmarks a while ago, copying a large file rather than a partition, between two disks (on the same disk, relative timings are different):
dd bs=64M 51.3
dd bs=1M 41.8
dd bs=4k 48.5
dd bs=512 48.9
cat 41.7
cp 45.3
The conclusion from this benchmark is that the choice of block size for dd
matters (but not that much), and cat
automatically finds the best way to make a fast copy: dd
can only slow you down. With a small block size, dd
wastes time making lost of tiny reads and writes. With a large block size, one disk remains idle while the other is reading or writing. The optimal rate is achieved when one disk reads while the other disk writes.
To copy a partition, it may be faster to copy the files with cp -a
. This depends on how many files there are and how much of the filesystem is free space. Copying files has an overhead that's roughly proportional to the number of files, but on the other hand copying free space wastes time.
The maximum data rate for USB2 is a little under 50 MB/s which works out to 6–7 hours to transfer 1TB. This assumes a hard disk that's fast enough to saturate the USB bus; I think the faster 7200 rpm drives can do it but 5900rpm might not be that fast (maybe they are for linear writes?).
If either disk is in use in parallel, this can slow down the copy considerably as the disk heads will need to move around.
The problem is your connection type, and block size. For the fastest results your block size should be half the lowest write speed you typically receive. This will give you a safe margin, but still allow for a large number; of course you need to have enough ram to hold the data too.
Usb 2.0 is 12 megabits per second (Mbps), Usb 2.0 High Speed is 480 Mbps. This is of course the raw speed; with 8 bits in a byte and framing overhead, the usable speed in MB/s is usually a decimal place over. So for example 480 raw, becomes 48MBs usable. Keep in mind that this is the mathematical best, in the real world it will be a bit lower. For usb 2.0 high speed connections you should expect somewhere around 30-35 MBs max write speed, provided the actual storage device can equate or surpass the connection speeds.