How to impose orthonormality constraints by method of Lagrange multipliers

The Lagrange multiplier method does put a special constraint on the structure of $\Lambda$, but that isn't what you expected.

From $\Phi\Phi^T=I$ and $R\Phi^T=\Phi^T\Lambda$, we obtain $\Lambda=\Phi\Phi^T\Lambda=\Phi R\Phi^T$. The constraint placed on the structure of $\Lambda$ is not that $\Lambda$ must be diagonal, but that $\Lambda$ must be symmetric.

Since $\Lambda$ is symmetric, it can be orthogonally diagonalised as $QDQ^T$. Therefore, $R\Phi^T=\Phi^T\Lambda$ implies that $R(\Phi^TQ)=(\Phi^TQ)D$. The eigenvectors of $R$ are the columns of $\Phi^TQ$ rather than the columns of $\Phi^T$.

This makes sense if you look at the original objective function. Since $\Phi R\Phi^T$ has the same trace as $(Q^T\Phi) R(\Phi^TQ)$ for every $Q\in SO(m)$, there is no reason why the columns of any $\Phi^T$ that minimises $\operatorname{tr}(\Phi R\Phi^T)$ must be eigenvectors of $R$.