What do the values of the mask parameter returned by findHomography represent?

The mask returned by findHomography is an 8-bit, single-channel cv::Mat (or std::vector<uchar>, if you prefer) containing either 0 or 1 indicating the outlier status.

EDIT: You access each element of the mask by calling .at<double>, which is leading to the confusing output. You should be using .at<uchar>, which will interpret the matrix value correctly.

I used the findHomography method after applying keypoint matching.

• Inliers are matched keypoints that are calculated to be true positives (correct matches);
• Outliers are matched keypoints that are calculated to be false positives (false matches).

Then you can use the mask output to extract the subset of correct matches from all matches.

• There is an example in Python 3.6 & OpenCV 3.4.1:

  good_kp = [gray_kp[m.queryIdx].pt for m in good_matches]
  correct_matched_kp = [good_kp[i] for i in range(len(good_kp)) if mask[i]]