Prove $R(a \times b) = Ra \times Rb$ given $R \in \mathcal{SO}(3)$ and $a,b \in \mathbb{R}^3$.

Recall that the cross product $a\times b$ is characterized by the property that

$$ \det(x,a,b)=\langle x,a\times b\rangle, \qquad \forall x\in\mathbb{R}^3. $$

Now let $R\in\mathcal{SO}(3)$. Then by using the fact that $R^{\mathsf{T}} = R^{-1}$, we get

$$ \langle x, R(a \times b) \rangle = \langle R^{\mathsf{T}}x, a \times b \rangle = \langle R^{-1}x, a \times b \rangle = \det(R^{-1}x, a, b). $$

Then, utilizing the assumption $\det(R) = 1$,

$$ = \det(R) \det(R^{-1}x, a, b) = \det(x, Ra, Rb) = \langle x, Ra \times Rb \rangle. $$

Finally, since $\langle x, R(a \times b) \rangle = \langle x, Ra \times Rb \rangle$ holds for any $x\in\mathbb{R}^3$, the desired identity follows.


Addendum. A similar argument shows that, for any invertible $3\times 3$ real matrix $T$,

$$ T(a \times b) = \frac{1}{\det T}(T T^{\mathsf{T}})( Ta \times Tb). $$


The simplest approach I see is the following, $$\langle R(a\times b), Ra\rangle = (Ra)^TR(a\times b)$$ $$=a^T(a\times b)=\langle a\times b, a\rangle =0$$ Similarly, $$\langle R(a\times b), Rb\rangle = 0$$ Hence, it follows that $$R(a\times b)= k( Ra\times Rb)$$ for some non-zero real number $k$. Now, observe that, $$\frac{1}{k}||R(a\times b)||^2$$ $$\langle R(a\times b), Ra\times Rb \rangle = det (R(a\times b), Ra,Rb)$$ $$=(det R)(det(a\times b,a,b))$$ $$=||a\times b||^2$$ $$=||R(a\times b)||^2$$ Required result follows.


I don't know whether this is really different from what you had in mind, but I would explain this as follows. It has a bit of Lie algebra air to it.

Let $R_{\vec{a},t}$ be the right-handed rotation about the axis $\vec{a}$ by the angle $t|\vec{a}|$. These form a 1-parameter group under composition, basically just add the angles of rotation as the axis is constant. The Lie algebra trick is to study the derivatives of such 1-parameter groups at $t=0$. The reason why cross products often appear when studying rotations in $\Bbb{R}^3$ is the following.

For all vectors $\vec{b}\in\Bbb{R}^3$ we have $$\frac d{dt}R_{\vec{a},t}\vec{b}\big\vert_{t=0}=\vec{a}\times\vec{b}.$$ In other words, cross product by $\vec{a}$ is an infinitesimal generator for the group of rotations about the axis $\vec{a}$.

The other fact I would use is the observation that

For all rotations $R$ we have the conjugation relation $$ R\circ R_{\vec{a},t}\circ R^{-1}=R_{R\vec{a},t}.$$ In other words, to rotate a vector $\vec{b}$ about the vector $R\vec{a}$ we can first apply $R^{-1}$, to move the scene to be about rotations about $\vec{a}$, then rotate about $\vec{a}$ by the prescribed angle, and then as the last step rotate the axis back to $R\vec{a}$.

We have the obvious identity $$ (R\circ R_{\vec{a},t}\circ R^{-1})R\vec{b}=R(R_{\vec{a},t}\vec{b}).\qquad(*) $$ And your claim follows from equating the derivatives of both sides of $(*)$ at $t=0$.


Don't know whether this was buried in one of your proofs, or whether you see too much handwavium in it.