What does Transform::linear() return in the Eigen library?

Having come across the same question, I would like to add the following information:

As explained by @ggael, Transform::linear() directly returns the linear part in the transformation matrix.

Transform::rotation() returns the rotational component in the linear part. But since the linear part contains not only rotation but also reflection, shear and scaling, extracting the rotation isn't straight forward, and needs to be calculated using a singular value decomposition (SVD).

In the common case where the affine matrix is known to contain only rotation and translation then Transform::linear() can be used to efficiently access the rotation part.

Finally, if you set the Transform's Mode template parameter to Isometry, Eigen will assume only rotations and translations, and optimize calculation of the inverse transform. The latest versions of Eigen will also optimize Transform::rotation() in that case, and directly return the linear part. Note, however, that there isn't a compact version of isometric affine matrices, where the last row isn't stored and assumed to be [0 ... 0 1].

The class Transform represents either an affine or a projective transformation using homogenous calculus. For instance, an affine transformation A is composed of a linear part L and a translation t such that transforming a point p by A is equivalent to:

 p' = L * p + t

Using homogeneous vectors:

 [p'] = [L t] * [p] = A * [p]
 [1 ]   [0 1]   [1]       [1]

with:

 A = [L t]
     [0 1]

So the linear part correspond to the top-left corner of the homogenous matrix representation. It corresponds to a mix of rotation, scaling and shearing.