Copying huge files between two remote machines - Efficiently

Solution 1:

This problem can be solved with rsync. At least this solution should be competitive in terms of performance.

First, rsync can be called from one of the remote systems to overcome the limitation in the inability to copy between two remote systems directly.

Second, encryption/decryption can be avoided by running rsync in Daemon Access mode instead of Remote Shell Access mode.

In daemon access mode rsync does not tunnel the traffic through an ssh connection. Instead it uses its own protocol on top of TCP.

Normally you run rsync daemon from inet.d or stand-alone. Anyway this requires root access to one of the remote systems. Assuming root access is not available, it is still possible to start up the daemon.

Start rsync daemon as a non-privileged user on the destination machine

ssh -i private_key ssh_user@destination-IP \
       "echo -e 'pid file = /tmp/rsyncd.pid\nport = 1873' > /tmp/rsyncd.conf

ssh -i private_key ssh_user@destination-IP \
       rsync --config=/tmp/rsyncd.conf --daemon

Actually copy the files

ssh -i private_key ssh_user@source_ip \
       "rsync [OPTIONS] source-path \
              rsync://ssh_user@destination-IP:1873:destination-path"

Solution 2:

The least-overhad solution would be using netcat:

destination$ nc -l -p 12345 > /path/destinationfile
source$ cat /path/sourcfile | nc desti.nation.ip.address 12345

(some netcat version do not need the "-p" flag for port)

All this does is send the unencrypted data, unauthenticated over the network from one pc to the other. Of course it is not the most "comfortable" way to do it.

Other alternatives would be trying to change the ssh cipher (ssh -c), or using ftp.

PS: rsync works fine with remote machines, but it is mostly used in combination with ssh, so no speedup here.

---

Solution 3:

If encryption isn't a concern, throw up an NFS daemon on C and mount the directory on B. Use rsync run on B, but specify the local directory paths.

Ignoring whatever your use case for involving A is, just prepend ssh user@B rsync... to the command.

Transfers data without encryption overhead and only transfers the different files.

Also, FTP was built with 3rd party server-to-server transfers as a protocol feature.