Speed up / avoid calculation of rsync file lists

As far as rsync is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous). If you can run an rsync on your remote server (so you get a true client-server scenario across your network) you'll gain a significant amount of traction.

Nevertheless here are other options to consider

  1. Copying without worrying about deleting old files. This may allow you to perform a faster transfer more often, and relegate the slower cleanup to say, just once a day:

    cp -au /mnt/ROUTER_WD_2TB/. /mnt/BACKUP_HITACHI_2TB/
    
  2. Use a variation of rsync's default deletion algorithm to avoid building a full list of files before the transfer:

    rsync -haAXi --quiet --delete --delete-during /mnt/ROUTER_WD_2TB/ /mnt/BACKUP_HITACHI_2TB/
    
  3. Split the top-level directories into separate tasks and run them in parallel. You may find that if you're disk IO-bound then this won't really help, and for spinning platters it almost certainly will make things worse.

    for d in /mnt/ROUTER_WD_2TB/*
    do
        rsync -haAXi --quiet --delete --delete-during "$d" /mnt/BACKUP_HITACHI_2TB/ >"/tmp/rsync.${d/*\/}.log" 2>&1 &
    done
    wait
    cat /tmp/rsync.*.log
    rm -f /tmp/rsync.*.log
    

If none of these suggestions helps then it would be worth adding another --verbose to rsync to see what it is doing. I suspect it's rattling through all the unchanged files, and if you have enough files this simply takes a long time.

Tags:

Backup

Rsync