rsync --delete --files-from=list / dest/ does not delete unwanted files
Perhaps you could do this using a list of include patterns instead, and use --delete-excluded
(which does as the name suggests)? Something like:
rsync -r --include-from=<patternlistfile> --exclude=* --delete-excluded / dest/
If filenames are likely to contain wildcard characters (*
, ?
and [
) then you may need to modify the Python to escape them:
re.sub("([[*?])", r"\\\1", "abc[def*ghi?klm")
Edit: Pattern-based matching works slightly differently to --files-from
in that rsync
won't recurse into directories that match the exclude pattern, for reasons of efficiency. So if your files are in /some/dir
and /some/other/dir
then your pattern file needs to look like:
/some/
/some/dir/
/some/dir/file1
/some/dir/file2
/some/other/
/some/other/dir/
/some/other/dir/file3
...
Alternatively, if all files are in the same directory then you could rewrite the command slightly:
rsync -r --include-from=<patternlistfile> --exclude=* --delete-excluded /some/dir/ dest/
and then your patterns become:
/file1
/file2
Edit: Thinking about it, you could include all directories with one pattern:
/**/
but then you'd end up with the entire directory tree in dest/
which probably isn't what you want. But combining it with -m
(which prunes empty directories) should solve that - so the command ends up something like:
rsync -m -r --delete-excluded --include-from=<patternfile> --exclude=* / dest/
and the pattern file:
/**/
/some/dir/file1
/some/other/dir/file3
As you explained, the command
rsync -r --delete --files-from=$FILELIST user@server:/ $DEST/
does not delete content in the destination when an entry from $FILELIST has been removed. A simple solution is to use instead the following.
mkdir -p $DEST
rm -rf $TEMP
rsync -r --link-dest=$DEST --files-from=$FILELIST user@server:/ $TEMP/
rm -r $DEST
mv $TEMP $DEST
This instructs rsync to use an empty destination. Files that are already present in the link-dest-directory are locally hard-linked and not copied. Finally the old destination is replaced by the new one. The first mkdir creates an empty $DEST if $DEST doesn't exist, to prevent rsync error. (The $-variables are assumed to carry the full path to the respective file or directory.)
There is some minor overhead for the hard-linking, but you don't need to mess with complex include/exclude-strategies.
Inspired from m4t, but using ... rsync for cleanup
rsync -r --link-dest=$dest --files-from=filelist.txt user@server:$source/ $temp
rsync -ra --delete --link-dest=$temp $temp/ $dest
This is not exactly the solution, but people coming here might find this useful: Since rsync 3.1.0 there is a --delete-missing-args
parameter which deletes files in the destination directory when you sync two directories using --files-from
. You would need to specify the deleted files in /tmp/list
along with files you do want copied:
rsync --delete-missing-args --files-from=/tmp/list /source/dir /destination/dir
See the man page for more details.