How to use Paramiko getfo to download file from SFTP server to memory to process it
A simple solution that still allows you to use progress callback is:
Use
BytesIO
file-like object to store a downloaded file to memory;You have to seek file pointer back to file start after downloading it, before you start reading it.
with io.BytesIO() as fl: sftp.getfo(file_name, fl, callback=printTotals) fl.seek(0) df = pd.read_csv(fl, sep=' ')
Though with this solution, you will end up having the file loaded to memory twice.
Better solution is to implement a custom file-like object. It will even allow you to download and parse the file at the same time.
class FileWithProgress:
def __init__(self, fl):
self.fl = fl
self.size = fl.stat().st_size
self.p = 0
def read(self, blocksize):
r = self.fl.read(blocksize)
self.p += len(r)
print(str(self.p) + " of " + str(self.size))
return r
And use it like:
with sftp.open(file_name, "rb") as fl:
fl.prefetch()
df = pd.read_csv(FileWithProgress(fl), sep=' ')
For the SFTPFile.prefetch
call, refer to:
Reading file opened with Python Paramiko SFTPClient.open method is slow.
What I ended up doing was a simple version of that, unfortunately without a callback for the progress, I also needed rb
for reading:
with sftp.open(file_name, 'rb') as fl:
df = pd.read_csv(fl, sep=' ')
Anyway, Martin's answer is what I was looking for!