matrix inequality with orthogonal matrices

I confirm Peter's suggestion that the best constant is $\frac1{\sqrt2}$. For let $\omega$ be this best constant. Then the inequality amounts to writing $$3-{\rm Tr}(A^tB^tAB)\le2\omega^2(3-{\rm Tr}\,A)(3-{\rm Tr}\,B).$$ Let me parametrize $A$ by the angle of rotation $\theta$ and the axis of rotation $u$, a unit vector. Likewise $B$ has an angle of rotation $\alpha$ and an axis $v$. The left-hand side above is $8\omega^2(1-c)(1-c')$, where $c=\cos\theta$ and $c'=\cos\alpha$.

Now $B^tAB$ is the rotation of same angle $\theta$ about $w=B^{-1}u$, and $A^t$ the rotation of angle $-\theta$ about $u$. It is an instructive calculus that $${\rm Tr}(A^tB^tAB)=2c+c^2+(1-c)^2(w\cdot u)^2+2s^2w\cdot u.$$ Because of $w\cdot u\ge c'$, we find that $\omega$ is the best constant in $$3-2c-c^2+(1-c)^2c'^2-2s^2c'\le8\omega^2(1-c)(1-c').$$ The left-hand side factorizes as $(1-c)(1-c')(3+c+c'-cc')$, and there remains the inequality $$3+c+c'-cc'\le8\omega^2.$$ Therefore $\omega^2$ is the supremum of $\frac18(3+c+c'-cc')$ over $[-1,1]^2$. Because this quantity if affine in both $c$ and $c'$, the supremum is achieved at a vertex, and its value is $4$. Whence $\omega^2=\frac48=\frac12$.

Numerical experiments suggest that actually $\|AB-BA\|_F\leq\frac{1}{\sqrt{2}} \|A-I\|_F\|B-I\|_F$ could be true, and that this bound is sharp.

I don't know if that helps, but this sharpened inequality is equivalent to \begin{equation} 3-\text{trace}(ABA^tB^t)\le(3-\text{trace}(A))(3-\text{trace}(B)). \end{equation}