MySQL High Performance for Lots of SELECTs/INSERTs/UPDATEs/DELETEs

If that's a bad solution would depend on many things. Does this data need to be persistent? Otherwise maybe a solution that simply keeps this data in memory would work better.

"A lot of users" isn't really helping anybody. MySQL will most likely be fine if "a lot" would mean a few hundreds. (Though depending on what else your database has to handle. Several thousand should most likely work too.)

After all, it doesn't matter that much, if you write those records to stay or delete them after a few seconds to minutes. Deleting just makes two operations out of one. And MySQL can for sure handle a very large amount of creating and deleting records. Make sure you use a simple index to find those records again for deletion.

But without actual numbers and some information about the hardware your database server uses, that can't be answered with much precision.

Best thing would be to write some small application, that simply simulates the amount of load you think you will get without doing much real processing, just drop a lot of records against the server, delete them, at the same rate run some queries like the rest of your program would generate. Have a look at your server and see, if that affects it in any way.

Not sure, but there are options to set for MySQL that would allow it to cache a table in memory completely.It does this anyway in many situationsand most likely you won't have to change much. But if you talk about a really very large amount of users and records, you can maybe tweak a few parameters to optimize caching for your special needs.


Here is a crazy idea. It involves assumptions and not always recommended practices (like updating a key) - I will get lots of negatives for suggesting this but here it goes...

Assuming that you have a very high volume of rows and high volume of deletes, you can enhance the delete performance by creating 2 partitions on your table. The partitions would differ by the first digit of the key. Example:

Key value 1123234441 is for active rows and key value: 9123234441 is for inactive rows (first digit in this example is used as follows: 1=active, 9=inactive).

Now when the user deletes a row, you don't physically delete the row, you update the key (Ouch!), this would automatically move the row to the inactive rows partition.

Of course, you need to restrict your selects to read data from the active partition only. Now the cool part is that dropping the inactive rows partition is extremely quick.

As I said earlier, this works if you have only 1 table. I have not tested this, so it is just a theoretical approach but I have experienced the speed of partition dropping and it is amazingly quick.

To enhance your selects, use proper indexing and to enhance inserts minimize row size and number of indexes (this statement is very generic...)

For a reference see: http://dev.mysql.com/doc/refman/5.1/en/partitioning-types.html Hope this helps.


One thing that has to be taken into consideration is how MySQL uses buffers for its major storage engines: InnoDB and MyISAM.

What lies cached in memory differs greatly between these storage engines.

InnoDB caches both data and index pages. They are loaded into the InnoDB Buffer Pool, which is sized by innodb_buffer_pool_size.

MyISAM caches only index pages and they are loaded into the Key Cache (Key Buffer), which is sized by key_buffer_size.

You must use information_schema.tables to get the data and index sizes occupied on disk in order to size the InnoDB Buffer Pool and MyISAM Key Cache correctly.

Depending on on how much data you have and how much time you will allow, you can warm the caches as follows:

For every table TableT

  • goto to each index NDX
  • for each index NDX
    • run SELECT every column in NDX,at least one column not indexed in TableT from TableT

By doing this you guarantee that every data and index page gets read at least once. They will sit in the cache. This concept is practiced, in part and in principle, by Percona. Percona built this concept into mk-slave-prefetch. What this program does is

  • read relay logs on a slave ahead of the slave processing the SQL in it
  • take an SQL statement from the relay log and convert it into a SELECT utilizing the WHERE, GROUP BY, and ORDER BY clauses as a guide to choosing indexes
  • execute the SELECT statement that came from converted SQL

This forces the slave to have 99.99% of the data needed by the slave to process the SQL quickly. This also makes the slave prepared in the event you manually failover to the slave and promote it to a master WHOSE CACHES ARE JUST ABOUT THE SAME AS THE MASTER YOU FAILED OVER FROM.

CONCLUSION

Nothing beats having caches ready, willing, and able for you to use in an environment of heavy INSERTS, UPDATEs, and DELETEs.

Give it a Try !!!

CAVEAT

With the birth of products like memcached, some have gotten away from the need to perform proper tuning of MySQL. Granted, many sites benefit from the boost in data retrieval provided by controlling the caching behavior of data as developers have quickly seen with memcached. Many other sites, just by switching storages engines or correctly configuring MySQL, have realized the same performance benefits. Before giving up on the database and strictly using it as a repository, make the most of the your database. Follow through on the due diligence and you might be pleasantly surprised what MySQL will do for you.

Tags:

Mysql

Performance

Recent Posts