How to prove this rotation identity in a coordinate-free way?

The "rotation for an angle $2\pi/3$ on $\mathbb R^2$" is usually defined using coordinates. If you want a coordinate-free proof for the mentioned statement, you must first clarify what a "rotation" on a $2$-dimensional real inner product space means. Depending on the definition, the proof will vary in difficulty. In the sequel, I suppose that $R$ is an orthogonal linear map (i.e. the inverse of $R$ is the adjoint of $R$ with respect to the given inner product) such that $R^2+R+I=0$.

Let us write $u,v,w$ for $u_1,u_2$ and $u_3$. The vector $u$ by definition is a unit vector. As $R$ is orthogonal, $v=Ru$ and $w=R^2u$ are also unit vectors. Since $R^2+R+I=0$, we have $u+v+w=0$. Thus \begin{aligned} \langle u,w\rangle+\langle v,w\rangle&=\langle u+v,w\rangle=\langle -w,w\rangle=-1,\\ \langle v,u\rangle+\langle w,u\rangle&=\langle v+w,u\rangle=\langle -u,u\rangle=-1,\\ \langle w,v\rangle+\langle u,v\rangle&=\langle w+u,v\rangle=\langle -v,v\rangle=-1.\\ \end{aligned} Therefore $\langle u,v\rangle=\langle v,w\rangle=\langle w,u\rangle=-\frac12$. Now let $g(f)=\frac23\left(\langle f,u\rangle u+\langle f,v\rangle v+\langle f,w\rangle w\right)$. Then $$ g(u)=\frac23\left(u-\frac12v-\frac12w\right) =\frac23\left(\frac32u-\frac{u+v+w}{2}\right)=u $$ and similarly, $g(v)=v$. However, since $x^2+x+1$ does not split over $\mathbb R$, the linear map $R$ has not any real eigenvalue. It follows that $au+bv=(aI+bR)u\ne0$ when $(a,b)\ne(0,0)$. Hence $\{u,v\}$ form a basis of $\mathbb R^2$ and $g(f)=f$ on this basis. In turn, we must have $g(f)=f$ on the whole vector space.


In terms of complex numbers, the inner product corresponds to

$$\langle\vec a,\vec b\rangle\leftrightarrow\Re(\overline ab)=\frac{\overline ab+a\overline b}{2}$$

so your equation becomes

$$\frac32f\overset?=u_1\frac{\overline u_1f+u_1\overline f}{2}+u_2\frac{\overline u_2f+u_2\overline f}{2}+u_3\frac{\overline u_3f+u_3\overline f}{2}$$

$$=\frac12\Big(u_1\overline u_1+u_2\overline u_2+u_3\overline u_3\Big)f+\frac12\Big(u_1\!^2+u_2\!^2+u_3\!^2\Big)\overline f$$

$$=\frac12\Big(|u_1|^2+|u_2|^2+|u_3|^2\Big)f+\frac12\Big(1+R^2+R^4\Big)u_1\!^2\overline f$$

$$=\frac12\Big(1+1+1\Big)f+\frac12\Big(0\Big)\overline f$$

$$=\frac32f$$

since $R^2=R^{-1}$ is a $3$rd root of unity, and any $n$'th root of unity $\zeta_n\neq1$ satisfies

$$1+\zeta_n+\zeta_n\!^2+\zeta_n\!^3+\cdots+\zeta_n\!^{n-1}=0$$

which can be seen by factoring

$$1-\zeta_n\!^n=(1-\zeta_n)(1+\zeta_n+\zeta_n\!^2+\zeta_n\!^3+\cdots+\zeta_n\!^{n-1}).$$


In terms of geometric algebra, the inner product is

$$\langle a,b\rangle=\frac{ab+ba}{2}$$

so your equation becomes

$$\frac32f\overset?=u_1\frac{u_1f+fu_1}{2}+u_2\frac{u_2f+fu_2}{2}+u_3\frac{u_3f+fu_3}{2}$$

$$=\frac12\Big(u_1u_1+u_2u_2+u_3u_3\Big)f+\frac12\Big(u_1fu_1+u_2fu_2+u_3fu_3\Big)$$

$$=\frac12\Big(\lVert u_1\rVert^2+\lVert u_2\rVert^2+\lVert u_3\rVert^2\Big)f+\frac12\Big(u_1fu_1+u_1Rfu_1R+u_1R^2fu_1R^2\Big),$$

where I've used $u_2=u_1R$ and $u_3=u_1R^2$; and the product of vectors $fu_1$ is a complex number (a "scalar" plus a "bivector"), so it commutes with the complex number $R$:

$$=\frac12\Big(\lVert u_1\rVert^2+\lVert u_2\rVert^2+\lVert u_3\rVert^2\Big)f+\frac12\Big(u_1fu_1+u_1fu_1R^2+u_1fu_1R^4\Big)$$

$$=\frac12\Big(1+1+1\Big)f+\frac12u_1fu_1\Big(1+R^2+R^4\Big)$$

$$=\frac32f+\frac12u_1fu_1\Big(0\Big)$$

by the same reasoning as before.