Sudden MySQL freeze, with spike in dirty pages, but without obvious write queries
You may need to tune your writes for InnoDB and possibly your ext4 volume
ASPECT #1
I noticed you have innodb_write_io_threads set to 4 (default value). You need to increase that so dirty pages can get flushed to their respective .ibd files more robustly. Please set it 16.
ASPECT #2
That pause to perform write might be do to ext4. Why ?
You have innodb_flush_method set to O_DIRECT. This is supposed to make disk writes more stable. Nevertheless, I a surprise for you. I answered ib_logfile opened with O_SYNC when innodb_flush_method=O_DSYNC about a year ago. I mentioned a Percona blog that says that O_DIRECT is faked in ext4 with the latest kernel.
- You need to make sure you have the latest Linux kernel (3.2 or greater)
- Change innodb_flush_log_at_trx_commit to 0
After a long experimentation, I've found the problem to be the infamous swap insanity.
The problem has been fixed by using the innodb-numa-interleave option.