rsync a list of directories with absolute path in text file
I want to use rsync to copy all these directories [from a list] preserving its absolute path to another location
Don't use --include
or --filter
variations, as that will just confuse things. Instead, use --files-from=_filename_
. (If you do, make sure you do lots of testing.)
Create the directory list in a file, one directory/file per line.
Use rsync's
--files-from=
with the above file.Use
--relative
/-R
option to make sure the source pathnames are copied at the end of the destination.Even if you have the
-a
option, also include-r
. From the man page:In both cases, if the -r option was enabled, that dir's entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a).
Complete command:
rsync ${DEBUG:+-nv} -arR --files-from=<list_of_files.txt> <top-level-dir> <target-dir>
(the files in list_of_files.txt must be relative or found in top-level-dir)
(if DEBUG is set, rsync merely prints out what might have been copied.)
Use the following command:
rsync -av --include-from=DirectoriesToCopy.txt --include /data/ --exclude='/data/*' --exclude='/*/' / /media/MyDestination/
You need to include /data/ explicitly, you could also have added that to the list in the file. Then exclude all other directories (order is important with includes/excludes).
Note that your usage of -r
was redundant as that's included in -a
.
EDIT: You could also accomplish the same result with:
rsync -av --relative /data/Dir1 /data/Dir2 /media/MyDestination/
It's not rsync that's forcing you to do difficult things just to copy a couple of directories, it just gives you multiple ways of doing the same thing; in some cases going the include/exclude way may be more suited, here I'd do the --relative thing above (without --relative
you'd end up with /media/MyDestination/Dir1
and /media/MyDestination/Dir2
, with the --relative
the whole source path is copied to the destination).
The rsync manual warns about this scenario (section “Include/exclude pattern rules”):
this won't work:
+ /some/path/this-file-will-not-be-found + /file-is-included - *
This fails because the parent directory
"some"
is excluded by the'*'
rule, so rsync never visits any of the files in the"some"
or"some/path"
directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule:"+ */"
(put it somewhere before the"- *"
rule), and perhaps use the--prune-empty-dirs
option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:+ /some/ + /some/path/ + /some/path/this-file-is-found + /file-also-included - *
In your case, I think the simplest approach would be to preprocess the list of directories to include so that whenever you include /path/to/foo
, you also include all the parent directories (/path/to
, /path
, /
), and also include subdirectories of the original directories (/path/to/foo/***
), and after all this have a rule that excludes everything not previously listed (*
).
<DirectoriesToCopy.txt awk '
{print "+ " $0 "/***"; while (sub(/\/+[^\/]+\/*$/, "/")) print "+ " $0}
END {print "- *"}
' >rsync-rules.txt
rsync -avr --include-from=rsync-rules.txt / /media/MyDestination/