How to insert values into a highly normalized database without excessive duplicate checking?
A little thought...
Formally you have something similar to this simplified structure:
CREATE TABLE slave1 (id PK, value UNIQUE);
CREATE TABLE slave2 (id PK, value UNIQUE);
CREATE TABLE main (id PK, id_slave1 FK, id_slave2 FK);
When you need to insert 2 records (id_1, val_1_1, val_1_2) and (id_2, val_2_1, val_2_2), you execute:
CREATE TEMPORARY TABLE temp (val_slave1, val_slave2) [ENGINE=Memory];
INSERT INTO temp (val_slave1, val_slave2)
VALUES (val_1_1, val_1_2),
(val_2_1, val_2_2);
INSERT IGNORE INTO slave1 (value)
SELECT DISTINCT val_slave1
FROM temp;
INSERT IGNORE INTO slave2 (value)
SELECT DISTINCT val_slave2
FROM temp;
INSERT INTO main (id_slave1, id_slave2)
SELECT slave1.id, slave2.id
FROM temp
JOIN slave1 ON temp.val_slave1 = slave1.value
JOIN slave2 ON temp.val_slave2 = slave2.value;
The engine of temp
may be Memory when the amount of inserted values is low, and InnoDB or something else if inserted data array is huge.
INSERT IGNORE works fast enough over UNIQUE indexed field. It guarantees that no duplicates in slave tables, and that the values which must be inserted will exist in slaves while insert into the main table.
And final query must be fast too - especially when temporary table fields are indexed too.
If you need to insert one record only, then you can, of course, do not use table temp
... but I think uniformity is more safe then a little simplification.
Of course, this may be optimized. For example, all inserts may be joined into one stored procedure, and you do not need in "60 database hits", one CALL is enough. Finally you must execute only 3 queries independent by the records count for to insert. And only one of them (inserting into temptable) may be huge (or even it can be divided to a lot of inserts).