Inserting only unique rows into SQLite (python)

One option is simply writing out the loop manually with an error catch instead of using executemany.

Pseudocode:

for row in csvfile:
   try:
       cursor.execute('INSERT INTO X (Y) VALUES (%s)' % row[rowdatapoint])
   except IntegrityError:
       pass

Probably not as efficient as executemany, but it will catch your error short of getting into more complicated SQL changes that would possibly involve you pregenerating a giant INSERT SQL string.

Simply use INSERT OR IGNORE to ignore the duplicates.

http://sqlite.org/lang_insert.html