Speed up / avoid calculation of rsync file lists
As far as rsync
is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous). If you can run an rsync
on your remote server (so you get a true client-server scenario across your network) you'll gain a significant amount of traction.
Nevertheless here are other options to consider
Copying without worrying about deleting old files. This may allow you to perform a faster transfer more often, and relegate the slower cleanup to say, just once a day:
cp -au /mnt/ROUTER_WD_2TB/. /mnt/BACKUP_HITACHI_2TB/
Use a variation of
rsync
's default deletion algorithm to avoid building a full list of files before the transfer:rsync -haAXi --quiet --delete --delete-during /mnt/ROUTER_WD_2TB/ /mnt/BACKUP_HITACHI_2TB/
Split the top-level directories into separate tasks and run them in parallel. You may find that if you're disk IO-bound then this won't really help, and for spinning platters it almost certainly will make things worse.
for d in /mnt/ROUTER_WD_2TB/* do rsync -haAXi --quiet --delete --delete-during "$d" /mnt/BACKUP_HITACHI_2TB/ >"/tmp/rsync.${d/*\/}.log" 2>&1 & done wait cat /tmp/rsync.*.log rm -f /tmp/rsync.*.log
If none of these suggestions helps then it would be worth adding another --verbose
to rsync
to see what it is doing. I suspect it's rattling through all the unchanged files, and if you have enough files this simply takes a long time.