Turn pandas dataframe into a file-like object in memory?
Python module io
(docs) has necessary tools for file-like objects.
import io
# text buffer
s_buf = io.StringIO()
# saving a data frame to a buffer (same as with a regular file):
df.to_csv(s_buf)
Edit. (I forgot) In order to read from the buffer afterwards, its position should be set to the beginning:
s_buf.seek(0)
I'm not familiar with psycopg2
but according to docs both copy_expert
and copy_from
can be used, for example:
cur.copy_from(s_buf, table)
(For Python 2, see StringIO.)
I had problems implementing the solution from ptrj.
I think the issue stems from pandas setting the pos of the buffer to the end.
See as follows:
from StringIO import StringIO
df = pd.DataFrame({"name":['foo','bar'],"id":[1,2]})
s_buf = StringIO()
df.to_csv(s_buf)
s_buf.__dict__
# Output
# {'softspace': 0, 'buflist': ['foo,1\n', 'bar,2\n'], 'pos': 12, 'len': 12, 'closed': False, 'buf': ''}
Notice that pos is at 12. I had to set the pos to 0 in order for the subsequent copy_from command to work
s_buf.pos = 0
cur = conn.cursor()
cur.copy_from(s_buf, tablename, sep=',')
conn.commit()