Full DD copy from hdd to hdd
dd
was useful in the old days when people used tapes (when block sizes mattered) and when simpler tools such as cat
might not be binary-safe.
Nowadays, dd if=/dev/sdb of=/dev/sdc
is a just complicated, error-prone, slow way of writing cat /dev/sdb >/dev/sdc
. While dd
still useful for some relatively rare tasks, it is a lot less useful than the number of tutorials mentioning it would let you believe. There is no magic in dd
, the magic is all in /dev/sdb
.
Your new command sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K
is again needlessly slow and complicated. The data is read 128kB at a time (which is better than the dd
default of 512B, but not as good as even larger values). It then goes through two pipes before being written.
Use the simpler and faster cat
command. (In some benchmarks I made a couple of years ago under Linux, cat
was faster than cp
for a copy between different disks, and cp
was faster than dd
with any block size; dd
with a large block size was slightly faster when copying onto the same disk.)
cat /dev/sdb >/dev/sdc
If you want to run this command in sudo
, you need to make the redirection happen as root:
sudo sh -c 'cat /dev/sdb >/dev/sdc'
If you want a progress report, since you're using Linux, you can easily get one by noting the PID of the cat
process (say 1234) and looking at the position of its input (or output) file descriptor.
# cat /proc/1234/fdinfo/0
pos: 64155648
flags: 0100000
If you want a progress report and your unix variant doesn't provide an easy way to get at a file descriptor positions, you can install and use pv
instead of cat
.
dd
uses a very small blocksize by default (512 bytes). That results in a lot of overhead (one read()
and write()
syscall for every 512 bytes).
It goes a lot faster when you use a larger blocksize. Optimal speeds start at bs=64k
or so. Most people use a still larger bs=1M
so it becomes human readable (when dd
says it copied 1234 blocks
, you know it's 1234 MiB
without doing any math). Using even larger blocksizes are unlikely to result in speed improvements, just higher memory consumption.
So the command should be:
dd bs=1M if=/dev/sdb of=/dev/sdc
If you already have a slow dd
running, you can interrupt it and resume with a faster dd
instance. For this it is important to know how far the copy progressed already. dd
usually prints the progress when you cancel it, or you can send it the USR1
signal while it is running to make it print its progress.
kill -USR1 $(pidof dd)
For example if it copied more than 1234MiB
, you can resume at position 1234MiB
using:
dd bs=1M seek=1234 skip=1234 if=/dev/sdb of=/dev/sdc
If it copied fewer than 1234MiB
, your copy will be incomplete. If it copied more than 1234MiB
, it will re-copy some already copied parts, which normally does not do any harm. So if in doubt you should pick a value slightly smaller than what you believe was already copied.
Getting statistics about ongoing dd process
You can use the kill
command with the appropriate signal to make dd
output statistics to standard error.
From the GNU dd man page:
Sending a USR1 signal to a running 'dd' process makes it print I/O statistics to standard error and then resume copying.
$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid
18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s
Make sure you check your man page for the correct signal first as it may differ on different dd implementations: (BSD dd uses SIGINFO).
Speeding up the process
- Connect each HDD to it's own SATA port so the data can be read from one device and written to the other at the same time.
- Use an appropriate blocksize using the
bs=
argument. Have a look at this thread on superuser and try some values for yourself. - Use separate
dd
invocations for reading and writing and use a pipe to connect them (dd if=/dev/sda bs=1M | dd of=/dev/sdb bs=1M
).
If you do this and specify a blocksize, make sure you use the same blocksize on each invocation. - You may try other optimizations like the
direct
argument. - Make sure your hard disks are not mounted or it may result in a corrupt copy.