EmguCV (OpenCV) ORBDetector finding only bad matches

I encountered the same problem and found a suitable solution this: github Emgu.CV.Example DrawMatches.cs in which everything works.

I modified the code and method FindMatch looks like that:

public static void FindMatch(Mat modelImage, Mat observedImage, out VectorOfKeyPoint modelKeyPoints, out VectorOfKeyPoint observedKeyPoints, VectorOfVectorOfDMatch matches, out Mat mask, out Mat homography)
{
    int k = 2;
    double uniquenessThreshold = 0.80;
    homography = null;
    modelKeyPoints = new VectorOfKeyPoint();
    observedKeyPoints = new VectorOfKeyPoint();
    using (UMat uModelImage = modelImage.GetUMat(AccessType.Read))
    using (UMat uObservedImage = observedImage.GetUMat(AccessType.Read))
    {
        var featureDetector = new ORBDetector(9000);
        Mat modelDescriptors = new Mat();
        featureDetector.DetectAndCompute(uModelImage, null, modelKeyPoints, modelDescriptors, false);
        Mat observedDescriptors = new Mat();
        featureDetector.DetectAndCompute(uObservedImage, null, observedKeyPoints, observedDescriptors, false);
        using (var matcher = new BFMatcher(DistanceType.Hamming, false))
        {
            matcher.Add(modelDescriptors);

            matcher.KnnMatch(observedDescriptors, matches, k, null);
            mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
            mask.SetTo(new MCvScalar(255));
            Features2DToolbox.VoteForUniqueness(matches, uniquenessThreshold, mask);

            int nonZeroCount = CvInvoke.CountNonZero(mask);
            if (nonZeroCount >= 4)
            {
                nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelKeyPoints, observedKeyPoints,
                    matches, mask, 1.5, 20);
                if (nonZeroCount >= 4)
                    homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(modelKeyPoints,
                        observedKeyPoints, matches, mask, 2);
            }
        }
    }
}

Using:

var model = new Mat(@"image0.jpg");
var scene = new Mat(@"image1.jpg");
Mat result = new Mat();
VectorOfKeyPoint modelKeyPoints;
VectorOfKeyPoint observedKeyPoints;
var matches = new VectorOfVectorOfDMatch();
Mat mask;
Mat homography;
FindMatch(model, scene, out modelKeyPoints, out observedKeyPoints, matches, out mask, out homography);
CvInvoke.WarpPerspective(scene, result, homography, model.Size, Inter.Linear, Warp.InverseMap);

Result:

If you want to watch the process use next code:

public static Mat Draw(Mat modelImage, Mat observedImage)
{
    Mat homography;
    VectorOfKeyPoint modelKeyPoints;
    VectorOfKeyPoint observedKeyPoints;
    using (VectorOfVectorOfDMatch matches = new VectorOfVectorOfDMatch())
    {
        Mat mask;
        FindMatch(modelImage, observedImage, out modelKeyPoints, out observedKeyPoints, matches, out mask, out homography);
        Mat result = new Mat();
        Features2DToolbox.DrawMatches(modelImage, modelKeyPoints, observedImage, observedKeyPoints,
            matches, result, new MCvScalar(255, 0, 0), new MCvScalar(0, 0, 255), mask);

        if (homography != null)
        {
            var imgWarped = new Mat();
            CvInvoke.WarpPerspective(observedImage, imgWarped, homography, modelImage.Size, Inter.Linear, Warp.InverseMap);
            Rectangle rect = new Rectangle(Point.Empty, modelImage.Size);
            var pts = new PointF[]
            {
                  new PointF(rect.Left, rect.Bottom),
                  new PointF(rect.Right, rect.Bottom),
                  new PointF(rect.Right, rect.Top),
                  new PointF(rect.Left, rect.Top)
            };

            pts = CvInvoke.PerspectiveTransform(pts, homography);
            var points = new Point[pts.Length];
            for (int i = 0; i < points.Length; i++)
                points[i] = Point.Round(pts[i]);

            using (var vp = new VectorOfPoint(points))
            {
                CvInvoke.Polylines(result, vp, true, new MCvScalar(255, 0, 0, 255), 5);
            }
        }
        return result;
    }
}

Using:

var model = new Mat(@"image0.jpg");
var scene = new Mat(@"image1.jpg");
var result = Draw(model, scene);

Result:

Solution

Problem 1

The biggest problem was actually a quite easy one. I had accidentally flipped my model and test descriptors when matching:

matcher.Add(imgTest.Descriptors);
matcher.KnnMatch(imgModel.Descriptors, matches, 1, null);

But if you look at the documentation of these functions you will see that you have to add the model(s) and match against the test image.

matcher.Add(imgModel.Descriptors);
matcher.KnnMatch(imgTest.Descriptors, matches, 1, null);

Problem 2

I don't know why by now but Features2DToolbox.GetHomographyMatrixFromMatchedFeatures seems to be broken and my homography was always wrong, warping the image in a strange way (similar to the above examples).

To fix this I went ahead and directly used the wrapper invoke to OpenCV FindHomography(srcPoints, destPoints, method). To be able to do this I had to write a little helper to get my data-structures in the right format:

public static Mat GetHomography(VectorOfKeyPoint keypointsModel, VectorOfKeyPoint keypointsTest, List<MDMatch[]> matches)
{
  MKeyPoint[] kptsModel = keypointsModel.ToArray();
  MKeyPoint[] kptsTest = keypointsTest.ToArray();

  PointF[] srcPoints = new PointF[matches.Count];
  PointF[] destPoints = new PointF[matches.Count];

  for (int i = 0; i < matches.Count; i++)
  {
    srcPoints[i] = kptsModel[matches[i][0].TrainIdx].Point;
    destPoints[i] = kptsTest[matches[i][0].QueryIdx].Point;
  }

  Mat homography = CvInvoke.FindHomography(srcPoints, destPoints, Emgu.CV.CvEnum.HomographyMethod.Ransac);

  //PrintMatrix(homography);

  return homography;
}

Results

Now everything works fine and as expected: