Is the recoverPose() function in OpenCV is left-handed?
According to the OpenCV document, the algorithm in the function recoverPose is based on the paper "Nistér, D. An efficient solution to the five-point relative pose problem, CVPR 2003." From equations in Section 2 in this paper, we know it uses the basic triangle relationship (see figure here):
x2 = R*x1 + t
Therefore, translation t is the vector from cam2 to cam1 in cam2 frame. This explains why you get the answer t close to [0; 0; -1].
Seems the recoverPose() function returns the first camera transform relatively to the second one (which was not intuitive for me, and is not clearly stated in the documentation). With this assumption test works correctly.