$AB-BA$ invertible and $A^2+B^2 = AB$ then $3$ divides $n$

This is an addendum to Sqyuli's answer that (hopefully) explains some of the motivation behind the solution. It's too long to phrase in a comment, so I'll add a bit of the sketch of that answer here (in slightly different notation)

1. Set $\omega$ be a primitive third root of unity (note that $\omega^2+\omega+1=0$).

2. Let $A+\omega B=M$, and consider $M\overline{M}$.

3. Show that $\det(M\overline{M})$ is real.

4. Use the second condition to reduce this to $\omega(BA-AB)$, and see

$$\det(\omega(BA-AB))=\omega^n\det(BA-AB)\in\mathbb{R}.$$

5. Conclude that $3\mid n$.

The key motivating step is to notice the factorization

$$a^2+ab+b^2=(a+\omega b)\left(a+\frac{1}{\omega} b\right),$$

which one can notice for example by noting $\frac{a^3-b^3}{a-b}=a^2+ab+b^2$. Unfortunately, this isn't applicable directly, as for one we certainly can't say that $A$ and $B$ commute (which would allow us to factor the second condition but invalidate the first). However, it does indicate somehow that $A\pm\omega B$ could be useful, and $A\pm\frac{1}{\omega}B$ as well via multiplication of the two. We attempt this, and see (remembering that $A$ and $B$ do not commute) that

\begin{align*} (A+\omega B)\left(A+\frac{1}{\omega} B\right) &=A^2+B^2+\omega(BA)+\frac{1}{\omega}(AB)\\ &=AB\left(1+\frac{1}{\omega}\right)+\omega(BA)\\ &=\omega(BA)-\omega(AB)\\ &=\omega(BA-AB), \end{align*}

from which the importance of the first condition becomes apparent.

This problem is an old problem of the IMC from 1997. You can find a solution of your question on their website.

The following is your problem divided in small parts, if you would like to try it by yourself (what I would suggest.)

1. Let $z = -\frac{1}{2}+i\frac{\sqrt{3}}{2} \in \mathbb{C}$ and define $M \in M_n(\mathbb{C})$ as $M = A+zB$
2. Show that $\det(M\overline{M}) \in \mathbb{R}$. [Hint: $z\overline{z} \in \mathbb{R}$]
3. Write $M\overline{M}$ as $z \cdot (...)$ and observe $\det(M\overline{M})=z^n\cdot \det(...)$. Also show that $\det(...) \neq 0$
4. Combine 2. and 3. to show that $3\mid n$.

Personally I don't know how anyone found $z$ and $M$ as defined above by solving this exercise. Perhaps someone else could explain this. In other words: I see that it works with $z$ and $M$ as defined above and I also understand the idea of the proof, but the definition of $z$ and $M$ appear out of nowhere (for me).