How can I create an in-memory database with sqlite?

I created a dataframe and dumped it into a memory db with a shared cache:

#sql_write.py
import sqlite3
import pandas as pd

conn = sqlite3.connect('file:cachedb?mode=memory&cache=shared')
cur  = conn.cursor()

df
          DT      Bid      Ask
0         2020-01-06 00:00:00.103000  1.11603  1.11605
1         2020-01-06 00:00:00.204000  1.11602  1.11605
...                              ...      ...      ...
13582616  2020-06-01 23:59:56.990000  1.11252  1.11256
13582617  2020-06-01 23:59:58.195000  1.11251  1.11255

[13582618 rows x 3 columns]


df.to_sql("ticks", conn, if_exists="replace")

Read from the memory db in another thread / script:

#sql_read.py
import sqlite3
import pandas as pd

conn = sqlite3.connect('file:cachedb?mode=memory&cache=shared')
cur  = conn.cursor()

df = pd.read_sql_query("select * from ticks", conn)

df
          DT      Bid      Ask
0         2020-01-06 00:00:00.103000  1.11603  1.11605
1         2020-01-06 00:00:00.204000  1.11602  1.11605
...                              ...      ...      ...
13582616  2020-06-01 23:59:56.990000  1.11252  1.11256
13582617  2020-06-01 23:59:58.195000  1.11251  1.11255

[13582618 rows x 3 columns]

Note that it's a 15-second read from in memory, on 1.35 million rows (python 2.7). If I pickle the same dataframe and open it, the read takes only 0.3 seconds: that was very disappointing to discover, as I was hoping to dump a huge table into memory and pull it up anywhere I wanted instantly. But there you go, pickle it is.

You create a new connection each time you call the function. Each connection call produces a new in-memory database.

Create the connection outside of the function, and pass it into the function, or create a shared memory connection:

db = sqlite3.connect("file::memory:?cache=shared")

However, the database will be erased when the last connection is deleted from memory; in your case that'll be each time the function ends.

Rather than explicitly call db.commit(), just use the database connection as a context manager:

try:
    with db:
        cur = db.cursor()
        # massage `args` as needed
        cur.execute(*args)
        return True
except Exception as why:
    return False

The transaction is automatically committed if there was no exception, rolled back otherwise. Note that it is safe to commit a query that only reads data.