Writing a BytesIO object to a file, 'efficiently'

shutil has a utility that will write the file efficiently. It copies in chunks, defaulting to 16K. Any multiple of 4K chunks should be a good cross platform number. I chose 131072 rather arbitrarily because really the file is written to the OS cache in RAM before going to disk and the chunk size isn't that big of a deal.

import shutil

myBytesIOObj.seek(0)
with open('myfile.ext', 'wb') as f:
    shutil.copyfileobj(myBytesIOObj, f, length=131072)

BTW, there was no need to close the file object at the end. with defines a scope, and the file object is defined inside that scope. The file handle is therefore closed automatically on exit from the with block.


Since Python 3.2 it's possible to use the BytesIO.getbuffer() method as follows:

from io import BytesIO
buf = BytesIO(b'test')
with open('path/to/file', 'wb') as f:
    f.write(buf.getbuffer())

This way it doesn't copy the buffer's content, streaming it straight to the open file.

Note: The StringIO buffer doesn't support the getbuffer() protocol (as of Python 3.9).

Before streaming the BytesIO buffer to file, you might want to set its position to the beginning:

buf.seek(0)

Tags:

Python

Io

Bytesio