Replace scp with sftp

Solution 1:

Rsync over sftp or chroot sftp

Edit: To be clear, the LFTP client supports numerous protocols, including FTP, HTTP, FISH, SFTP, HTTPS, FTPS and BitTorrent. In this example, we are using SFTP. The client name has caused some confuion from folks reading this post.

You can use lftp and it's mirror sub-system to replicate the behavior of rsync over SFTP. This works in chroot environments as well.

lftp -u username, \
-e "set net:timeout 4;set net:max-retries 6;mirror -R --parallel=8 --no-perms --newer-than=now-2days --only-newer /source/ /source/;bye" sftp://hostname.domain

In this example, I left out the password, because we have an SSH key trust to the destination host.

Note: The -R does not mean recursive. The -R flag means push to remote.

--parallel= is the number of threads to break the job up into. This can be limited or impacted of course by MaxStartups in /etc/ssh/sshd_config and nofile pam limits and other limits on the target server.

net:max-retries 6 I like to retry, just in case there was a network maintenance.

net:timeout 4 give up, then retry after 4 seconds.

--no-perms if you don't wish to change perms on the remote end.

--newer-than is just an example of something to use as needed.

--loop can be used to repeat the sync until there are not changes detected.

bye we don't want to forget to log out instead of relying on idle timeout.

There are many other options available in LFTP.

I have had great experiences with LFTP using it's mirror sub-system and SFTP. It is substantially faster, as it can break jobs up into multiple streams. It can even break up a single file into multiple streams.

A major security advantage of using LFTP is allowing file transfers in environments where you don't want the automation scripts to have access to a shell on the remote end.

I created a working demo that you are welcome to test with. File retention is low, but you are welcome to use this to see how easy it is to automate pushing directories / files using lftp plus sftp chroot. Simply install lftp and then create some temp dirs or files, then:

mkdir -p ~/sftp_test/`date '+%Y%m%d%H%M'`/{1,2,3}/{a,b,c}
lftp -u anon, -e "set net:timeout 4;set net:max-retries 4;mirror -R --parallel=4 --no-perms ~/sftp_test/ /private/;bye" sftp://ohftp.org

Solution 2:

You could use a batchfile aproximation:

sftp -b batchfile [user@]host

a batchfile includes a sequence of commands that must reproduce "scp" operation with equivalent ftp commands:

cd remote_dir
lcd local_dir
put file