Writing a BytesIO object to a file, 'efficiently'
shutil
has a utility that will write the file efficiently. It copies in chunks, defaulting to 16K. Any multiple of 4K chunks should be a good cross platform number. I chose 131072 rather arbitrarily because really the file is written to the OS cache in RAM before going to disk and the chunk size isn't that big of a deal.
import shutil
myBytesIOObj.seek(0)
with open('myfile.ext', 'wb') as f:
shutil.copyfileobj(myBytesIOObj, f, length=131072)
BTW, there was no need to close the file object at the end. with
defines a scope, and the file object is defined inside that scope. The file handle is therefore closed automatically on exit from the with
block.
Since Python 3.2 it's possible to use the BytesIO.getbuffer() method as follows:
from io import BytesIO
buf = BytesIO(b'test')
with open('path/to/file', 'wb') as f:
f.write(buf.getbuffer())
This way it doesn't copy the buffer's content, streaming it straight to the open file.
Note: The StringIO
buffer doesn't support the getbuffer()
protocol (as of Python 3.9).
Before streaming the BytesIO
buffer to file, you might want to set its position to the beginning:
buf.seek(0)