How does RSYNC do incremental Backups
Solution 1:
Rsync has a number of flags which control what it will look at and what it will copy over to the destination. Most often the "-a" flag is used which is the "Archive" flag, this is probably what you want. run rsync with the "-av" flags and have it do a first run against the data you want backed up. The next time it runs it'll do a block checksum of the file, and only copy over the parts which have been modified on existing files, copy new files over, and remove files which are no longer there. Check the "-a" options section on:
http://linux.die.net/man/1/rsync
The first run will be BandWidth intensive, the following runs will most likely be processor intensive but use little Bandwidth compared to the initial run. Unless you have a lot of churn over your data set.
Rsync doen't care how you got the files in the source, or the destination directories, it's only going to copy the changes between the two, unless you add flags to do something different.
If you want to log what was changed you can use the "--log-file" option. All in all something like this sounds like what you want:
rsync -av --log-file=/var/log/rsync.log -e "ssh -l backup-user" backup-user@source-machine::module /nas01/backups
Solution 2:
rsync doesn't do 'incremental', it's more like 'differential'. it doesn't transfer changes (which assume some knowledge of a prior run), it transfers differences (by comparing the source with the target files)
a simplification of the process:
- first checks file size, creation/modification dates, flags... if it's all identical, skip the file.
- if there's no file with that name on the target, simply copies the whole file.
- if there's a file on the target, it calculates checksums for each 2KB of the file and transfers to the sender.
- the sender compares the content of the origin file and compares with the target, and transfers any data not already there, together with references to any matched data. with that, the target can reconstruct the whole file using pieces of the old target and new blocks.
Solution 3:
Maybe I being pedantic but incremental backups mean you have a full backup first. Then you have backup of files changed since that backup. Then you have another backup of the backups changed since the previous one, etc. So you need the full backup, and all incremental backups since that one.
So, just using archive mode is not a incremental backup. I think the difference is important because it means you can't go back in time to get files before they changed.
If you want to do a true incremental backup, you use options such as --backup-dir. There is an example here.
Solution 4:
rsync does not log any data, it checks file modification timestamps and then content. if you upload by ftp first it'll be fine - rsync will not re-transmit all data but probably will go through all the content and fix timestamps. but there will be no huge transfer again.