How can I parallelise the upload of a directory by FTP?
lftp would do this with the command mirror -R -P 20 localpath
- mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once.
As explained in man lftp
:
mirror [OPTS] [source [target]]
Mirror specified source directory to local target directory. If target
directory ends with a slash, the source base name is appended to target
directory name. Source and/or target can be URLs pointing to directo‐
ries.
-R, --reverse reverse mirror (put files)
-P, --parallel[=N] download N files in parallel
You could try using gnu parallel and curl to automate it.
then you could do something such as:
find . -t f -name "*.pdf" | parallel -j 4 curl -T {} ftp://ftp.site.com --user me:pass
This will run 4 jobs per cpu uploading all pdfs in working path.