OpenCV - Calibrate fisheye lens error (Ill-conditioned matrix)

I did not find the code in python so I manually check the images with chessboard on the edge and delete them one by one until the error is gone.

I think it is because your variable calibration_flags has CALIB_CHECK_COND set. Try disabling this flag. Without it I was able to undistort your images (see links below).

I am not sure what this check is for (the documentation is not very explicit). This flag reject some images¹ of my gopro hero 3 even when the chessboard is visible and detected. In my case one image among 20 is not passing this test. This image has the chessboard close to the left border.

• https://i.stack.imgur.com/m2WF6.jpg
• https://i.stack.imgur.com/KiTRz.jpg
• https://i.stack.imgur.com/MhHyN.jpg
• https://i.stack.imgur.com/pSiyG.jpg
• https://i.stack.imgur.com/drXSL.jpg
• https://i.stack.imgur.com/DDze1.jpg
• https://i.stack.imgur.com/b6l8f.jpg
• https://i.stack.imgur.com/9MrAk.jpg
• https://i.stack.imgur.com/WYmg5.jpg
• https://i.stack.imgur.com/rmJ5Q.jpg
• https://i.stack.imgur.com/K8k8Y.jpg
• https://i.stack.imgur.com/MiBER.jpg

¹ in OpenCV versions >= 3.4.1 the error message tells you which image is not passing the test

As @Ahmadiah mentioned, the "ill conditioned" thing can happen when the checkerboard falls near the edge of the image. One way to handle this is to remove images one by one and try again when they cause calibration to fail. Here is an example where we do that:

def calibrate_fisheye(all_image_points, all_true_points, image_size):
    """ Calibrate a fisheye camera from matching points.
    :param all_image_points: Sequence[Array(N, 2)[float32]] of (x, y) image coordinates of the points.  (see  cv2.findChessboardCorners)
    :param all_true_points: Sequence[Array(N, 3)[float32]] of (x,y,z) points.  (If from a grid, just put (x,y) on a regular grid and z=0)
        Note that each of these sets of points can be in its own reference frame,
    :param image_size: The (size_y, size_x) of the image.
    :return: (rms, mtx, dist, rvecs, tvecs) where
        rms: float - The root-mean-squared error
        mtx: array[3x3] A 3x3 camera intrinsics matrix
        dst: array[4x1] A (4x1) array of distortion coefficients
        rvecs: Sequence[array[N,3,1]] of estimated rotation vectors for each set of true points
        tvecs: Sequence[array[N,3,1]] of estimated translation vectors for each set of true points
    """
    assert len(all_true_points) == len(all_image_points)
    all_true_points = list(all_true_points)  # Because we'll modify it in place
    all_image_points = list(all_image_points)
    while True:
        assert len(all_true_points) > 0, "There are no valid images from which to calibrate."
        try:
            rms, mtx, dist, rvecs, tvecs = cv2.fisheye.calibrate(
                objectPoints=[p[None, :, :] for p in all_true_points],
                imagePoints=[p[:, None, :] for p in all_image_points],
                image_size=image_size,
                K=np.zeros((3, 3)),
                D=np.zeros((4, 1)),
                flags=cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC + cv2.fisheye.CALIB_CHECK_COND + cv2.fisheye.CALIB_FIX_SKEW,
                criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-6),
            )
            print('Found a calibration based on {} well-conditioned images.'.format(len(all_true_points)))
            return rms, mtx, dist, rvecs, tvecs
        except cv2.error as err:
            try:
                idx = int(err.message.split('array ')[1][0])  # Parse index of invalid image from error message
                all_true_points.pop(idx)
                all_image_points.pop(idx)
                print("Removed ill-conditioned image {} from the data.  Trying again...".format(idx))
            except IndexError:
                raise err