Show that if $\Omega_1,\Omega_2$ are tangent vectors of curves in $SO(3)$ then $[\Omega_1,\Omega_2]\in\mathfrak{so}(3)$

We want to show that $$ \tag{1}[\Omega_1,\Omega_2] = \left.\frac{d}{dt}R_1(t)\Omega_2 R_1(t)^{-1}\right|_{t=0} $$

We can start doing this by computing: \begin{eqnarray*} \left.\frac{d}{dt}\left(R_1(t)\Omega_2 R_1(t)^{-1}\right)\right|_{t=0} &=& \left.\left[\frac{d}{dt}R_1(t)(\Omega_2 R_1(t)^{-1})\right]\right|_{t=0} + \left.\left[(R_1(t)\Omega_2) \frac{d}{dt}R_1(t)^{-1}\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] + \left.\left[R_1(t)\Omega_2 \frac{d}{dt}R_1(t)^{-1}\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] + \left.\left[R_1(t)\Omega_2 (-R_1(t)^{-1})\left(\frac{d}{dt}R_1(t)\right)R_1(t)^{-1})\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left.\left[R_1(t)\Omega_2 R_1(t)^{-1}\left(\frac{d}{dt}R_1(t)\right)R_1(t)^{-1})\right]\right|_{t=0} \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left[R_1(0)\Omega_2 R_1(0)^{-1}\left.\left(\frac{d}{dt}R_1(t)\right|_{t=0}\right)R_1(0)^{-1})\right] \\ &=& \left[\Omega_1\Omega_2 R_1(0)^{-1}\right] - \left[R_1(0)\Omega_2 R_1(0)^{-1}\Omega_1R_1(0)^{-1})\right] \\ \end{eqnarray*}

See Derivatives of Inverse Matrix for details on the third line's derivation.

There isn't much further we can go without knowing more details about $R_1$. I actually think it might have been defined as $$R_i(t) = e^{\Omega_it}$$ But it's only necessary to require $R_i(0) = I$. Then the final line above gives $\Omega_1\Omega_2 - \Omega_2\Omega_1$.

---

Update to include additional requirements:

The curve is in $SO(3)$

This is not always true. A Lie group is a subgroup of $GL$ but a Lie algebra always contains the $0$ matrix. It maps to the identity of the Lie group. So if $\Omega_2 = 0$ in the above, then the curve $C(t) = 0$ does not lie in $SO(3)$.

Show $[\Omega_1, \Omega_2]$ is in $\mathfrak{so}(3)$

This is true independently of the above considerations. $\Omega_1, \Omega_2$ are elements of $\mathfrak{so}(3)$ which is closed under addition and multiplication (matrices of the form $X^T = -X$).

Formula (1) is a bit strange indeed, $R_1(t)\Omega_2 R_1(t)^{-1}$ is not necessarily in $SO(3)$ ($\Omega_2$ could have a null determinant for example).

One way to prove the result, is to consider that (for $t \to 0$):

$$ R_1(t) = I + t \Omega_1 + t^2 W_1 + o(t^2)$$ $$ R_2(t) = I + t \Omega_2 + t^2 W_2 + o(t^2)$$

Then:

$$ R_1(t)R_2(t) = I + t (\Omega_1 + \Omega_2) + t^2 (W_1 + W_2 + \Omega_1 \Omega_2) + o(t^2)$$ $$ R_2(t)R_1(t) = I + t (\Omega_1 + \Omega_2) + t^2 (W_1 + W_2 + \Omega_2 \Omega_1) + o(t^2)$$

The you can consider $L(t) = R_1(t)^{-1}R_2(t)^{-1}R_1(t)R_2(t)$ which is a curve in $SO(3)$. Since $R_2(t)R_1(t)L(t) = R_1(t)R_2(t)$, you can develop term by term each side and find that:

$$ L(t) = I + t^2([\Omega_1, \Omega_2]) + o(t^2)$$

If you re-parametrise L with $s = t^2$ you have $L(s) = I + s([\Omega_1, \Omega_2]) + o(s)$ Thus you have found a curve in $SO(3)$ whose tangent is $[\Omega_1, \Omega_2]$.