Detecting garbage homographies from findHomography in OpenCV?

There are several sanity tests you can perform on the output. On top of my head:

1. Compute the determinant of the homography, and see if it's too close to zero for comfort.
2. Even better, compute its SVD, and verify that the ratio of the first-to-last singular value is sane (not too high). Either result will tell you whether the matrix is close to singular.
3. Compute the images of the image corners and of its center (i.e. the points you get when you apply the homography to those corners and center), and verify that they make sense, i.e. are they inside the image canvas (if you expect them to be)? Are they well separated from each other?
4. Plot in matlab/octave the output (data) points you fitted the homography to, along with their computed values from the input ones, using the homography, and verify that they are close (i.e. the error is low).

A common mistake that leads to garbage results is incorrect ordering of the lists of input and output points, that leads the fitting routine to work using wrong correspondences. Check that your indices are correct.

Understanding the degenerate homography cases is the key. You cannot get a good homography if your points are collinear or close to collinear, for example. Also, huge gray squares may indicate extreme scaling. Both cases may arise from the fact that there are very few inliers in your final homography calculation or the mapping is wrong.

To ensure that this never happens:
1. Make sure that points are well spread in both images.
2. Make sure that there are at least 10-30 correspondences (4 is enough if noise is small).
3. Make sure that points are correctly matched and the transformation is a homography.

To find bad homographies apply found H to your original points and see the separation from your expected points that is |x2-H*x1| < Tdist, where Tdist is your threshold for distance error. If there are only few points that satisfy this threshold your homography may be bad and you probably violated one of the above mentioned requirements.