"Invalid parameter type" (numpy.int64) when inserting rows with executemany()
I did the same as Robert; I converted everything to string. In my case, it was a pandas data frame that I casted to string type:
data = pandas.read_json(...)
data.astype(str).to_sql(...)
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html
If the data you are retrieving includes URLs, you may get a "unknown protocol" error (or something like that). If you get this error even after casting to string type, try using StringIO
instead:
import requests
from io import StringIO
...
data = pandas.read_json(StringIO(response.text))
where response
is an instance of object Response
from the requests
library and its attribute text
contains the json text data.
Your problem is not with the volume of data per se, it is that some of your tuples contain numpy.int64
values that cannot be used directly as parameter values for your SQL statement. For example,
a = numpy.array([10, 11, 12], dtype=numpy.int64)
params = (1, 1, a[1], 1, 1, 1)
crsr.execute(sql, params)
will throw
ProgrammingError: ('Invalid parameter type. param-index=2 param-type=numpy.int64', 'HY105')
because the third parameter value is a numpy.int64
element from your numpy array a
. Converting that value with int()
will avoid the issue:
a = numpy.array([10, 11, 12], dtype=numpy.int64)
params = (1, 1, int(a[1]), 1, 1, 1)
crsr.execute(sql, params)
By the way, the reason that
sql = 'SET GLOBAL max_allowed_packet=50*1024*1024'
cursor.execute(sql)
didn't work is that max_allowed_packet
is a MySQL setting that does not have any meaning for Microsoft SQL Server.