ZFS on FreeBSD: recovery from data corruption

Solution 1:

The problem was that the new motherboard's BIOS created a host protected area (HPA) on some of the drives, a small section used by OEMs for system recovery purposes, usually located at the end of the harddrive.

ZFS maintains 4 labels with partition meta information and the HPA prevents ZFS from seeing the upper two.

Solution: Boot Linux, use hdparm to inspect and remove the HPA. Be very careful, this can easily destroy your data for good. Consult the article and the hdparm man page (parameter -N) for details.

The problem did not only occur with the new motherboard, I had a similar issue when connecting the drives to an SAS controller card. The solution is the same.

Solution 2:

The very first thing I would recommend you do is to get some more hard drives and make duplicate copies of the 8 drives you have with your data on them, using the dd command. That way, if in your attempts to recover them you end up making things worse, you can still go back to this baseline.

I've done this before and there were times I didn't need it, but the times I did need it made it totally worth the effort.

Don't work without a net.

---

Solution 3:

You seem to be on track to solving this. If you want another, possible newer point of view you could try a Solaris 11 Express live CD. There's likely a lot newer code running there (zpool in Solaris is now at version 31, whereas you are at version 6) and it might offer better recovery possibilities. Don't run zpool upgrade under Solaris though if you want to keep the pool mountable under FreeBSD.