CSV file upload from buffer to S3

Have you tried calling buff.flush() first? It's possible that your entirely-sensible debugging check (calling getvalue()) is creating the illusion that the buff has been written to, but isn't if you don't call it.


Okay, disregard my earlier answer, I found the actual problem.

According to the boto3 documentation for the upload_fileobj function, the first parameter (Fileobj) needs to implement a read() method that returns bytes:

Fileobj (a file-like object) -- A file-like object to upload. At a minimum, it must implement the read method, and must return bytes.

The read() function on a _io.StringIO object returns a string, not bytes. I would suggest swapping the StringIO object for a BytesIO object, adding in the necessary encoding and decoding.

Here is a minimal working example. It's not the most efficient solution - the basic idea is to copy the contents over to a second BytesIO object.

import io
import boto3
import csv

buff = io.StringIO()

writer = csv.writer(buff, dialect='excel', delimiter=',')
writer.writerow(["a", "b", "c"])

buff2 = io.BytesIO(buff.getvalue().encode())

bucket = 'changeme'
key = 'blah.csv'

client = boto3.client('s3')
client.upload_fileobj(buff2, bucket, key)

As explained here using the method put_object rather than upload_fileobj would just do the job right with io.STRINGIO object buffer.

So here, to match the initial example:

client = boto3.client('s3')
client.upload_fileobj(buff2, bucket, key)

would become

client = boto3.client('s3')
client.put_object(Body=buff2, Bucket=bucket, Key=key, ContentType='application/vnd.ms-excel')