MySQL InnoDB crash post-mortem

I have good news, and bad news. The good news is, your filesystem and mysql are most likely fine... but check /var/log/syslog or equivalent to see what else was happening on your system before 10:21:05.

When the first message you posted was logged, your mysql server had already died.

120927 10:21:05 mysqld_safe Number of processes running now: 0

So, assuming you didn't overlook anything in the mysql error log, I'm going to say it didn't crash and die -- it was actually killed.

When mysqld_safe (which is a wrapper, not the server itself) realized the server wasn't running, and that the server hadn't terminated gracefully, it restarted it for you...

120927 10:21:06 mysqld_safe mysqld restarted

...then the server daemon logged some normal startup messages...

120927 10:21:12 [Note] Plugin 'FEDERATED' is disabled.
120927 10:21:12 InnoDB: The InnoDB memory heap is disabled
120927 10:21:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120927 10:21:12 InnoDB: Compressed tables use zlib 1.2.3
120927 10:21:12 InnoDB: Using Linux native AIO

...but when mysqld asked the operating system to allocate 4GB of memory for the InnoDB buffer pool...

120927 10:21:13 InnoDB: Initializing buffer pool, size = 4.0G

...the kernel said "no."

InnoDB: mmap(4395630592 bytes) failed; errno 12

Checking the kernel source to be sure:

#define ENOMEM      12  /* Out of memory */

Yeah. So, every message below the "failed; errno 12" line should be disregarded -- they're all side-effects of this one.

But again, all of these things happened after the first crash.

My best guess is that an extreme low memory condition caused your kernel to originally kill mysqld in an attempt to stablize the system.

Naturally, whatever caused the memory shortage was gone after the reboot. The mysql server was able to allocate 4GB for the InnoDB buffer pool, and all should be good until whatever caused you to run out of memory causes it again.

First guess: apache child processes run amok.


This happened to me recently and this thread has been invaluable for helping me understand what's happening.

I'm running a LAMP stack on Ubuntu 14.04 with 1GB Ram. My server kept crashing due to web traffic spikes. For me, fiddling with the mysql config file only extended the amount of time I'd experience another random crash. To test and fix this, I ultimately used Apache's ab tool:

ab -n 100 -c 10 http://gastonia.com/

10 concurrencies (-c) was ok, so incremented that until I hit 30 - bam - crash. I backed off until I found a number that was safe, and then I adjusted Apache's ServerLimit directive:

ServerLimit 20

After that I could change -c to any number I wanted to, and I have yet to experience another crash.

Hopeully that is helpful to anyone experience the same problem.