How to copy multiple files simultaneously using scp

You can use background task with wait command. Wait command ensures that all the background tasks are completed before processing next line. i.e echo will be executed after scp for all three nodes are completed.

#!/bin/bash
scp -i anuruddha.pem myfile1.tar [email protected]:/tmp &
scp -i anuruddha.pem myfile2.tar [email protected]:/tmp &
scp -i anuruddha.pem myfile.tar [email protected]:/tmp &

wait
echo "SCP completed"

SSH is able to do so-called "multiplexing" - more connections over one (to one server). It can be one way to afford what you want. Look up keywords like "ControlMaster"

Second way is using more connections, then send every job at background:

for file in file1 file2 file3 ; do 
     scp $file server:/tmp/ & 
done

But, this is answer to your question - "How to copy multiple files simultaneously". For speed up, you can use weaker encryption (rc4 etc) and also don't forget, that the bottleneck can be your hard drive - because SCP don't implicitly limit transfer speed.

Last thing is using rsync - in some cases, it can be lot faster than scp...


I am not sure if this helps you, but I generally archive (compression is not required. just archiving is sufficient) file at the source, download it, extract them. This will speed up the process significantly. Before archiving it took > 8 hours to download 1GB After archiving it took < 8 minutes to do the same

Tags:

Linux

Scp