How do you identify InnoDB table corruption?
Morgan gives a hint in his comment that InnoDB is constantly checking for corrupted pages by doing checksums on the pages it reads. If InnoDB finds a checksum mismatch, it will crash stop the server.
If you want to speed that process up (instead of waiting for InnoDB to read the corrupted page), you can use innochecksum
:
Because checksum mismatches will cause InnoDB to deliberately shut down a running server, it can be preferable to use this tool rather than waiting for a server in production usage to encounter the damaged pages.
An interesting caveat:
innochecksum cannot be used on tablespace files that the server already has open. For such files, you should use CHECK TABLE to check tables within the tablespace.
So yes, for an online table CHECK TABLE
is probably the tool (or as pointed out in another answer mysqlcheck
if you want to do more than a single database at a time.)
If you can shut down your database you can force it the checksums using innochecksum
Anecdotal:
On a innodb tablespace of 29GB (with innodb_file_per_table=1
), this script took about 2 Minutes
#!/bin/bash
for i in $(ls /var/lib/mysql/*/*.ibd)
do
innochecksum -v $i
done
As a bonus, though, since you are running Percona, they implemented a new method for fast innodb checksum. I've never used it, but it might speed up the process.
WARNING: before trying any of these instructions it is strongly recommended to verify that there is a healthy backup of your database in hands, just in case. (thanks to @Nick for the warning)
Try to use the mysqlcheck
command. On a terminal:
mysqlcheck -u username -p --databases database1 database2
This command will output a list of all tables and a status telling you if there was some kind of corruption:
table1 OK
table2 OK
table3 OK
tableN OK
With that in hands you will already know which tables you have to repair. Just in case you want to repair everything at once:
mysqlcheck -u username -p --auto-repair --databases database1 database2
More about mysqlcheck
: http://dev.mysql.com/doc/refman/5.0/en/mysqlcheck.html
Note: you tagged your question with percona. I had no clue of what was that so I googled. It seems to be a fork of MySQL, but I have no reason to believe the commands are incompatible (fingers crossed).
Somebody pointed me to this guide that has more specific instructions for InnoDB database recovery for more critical situations where the entire database does not start: http://www.softwareprojects.com/resources/programming/t-how-to-fix-mysql-database-myisam-innodb-1634.html
According to MySQL 5.0 Certification Study Guide, Page 443,444 Section 30.4:
You can check InnoDB tables by using the CHECK TABLE command or using a client program to issue the statement for you. However, if an InnoDB table has problems, you cannot fix it by using REPAIR TABLE because that statement applies only to MyISAM.
If a table check indicates that an InnoDB table has problems, you should be able to restore the table to a consistent state by dumping it with mysqldump, dropping it, and recreating it from that dump.
In the event of a crash of a MySQL Server or on the host on which it runs, some InnoDB tables might need repairs. Normally, it suffices simply to restart the server because the InnoDB storage engine performs auto-recovery as part of its startup sequence. In rare cases, the server might not start up due to the failure of InnoDB auto-recovery. If that happens, use the following procedure:
Restart the server with the --innodb_force_recovery option set to a value in the rage from 1 to 6. These values indicate increasing levels of caution in a avoiding a crash, and increasing levels of tolerance for possible inconsistency in the recovered tables. A good value to start with is 4.
When you start the server with --innodb_force_recovery set to a non-zero value, InnoDB treats the tablespace as read-only. Consequently, you should dump the InnoDB tables with mysqldump and then drop them while the option is in effect. Then restart the server without the --innodb_force_recovery option. When the server comes up, recover the InnoDB tables from the dump files.
If the preceding steps fail, it's necessary to restore the InnoDB tables from a previous backup.
Please read MySQL Docs on InnoDB Forced Recovery