Prove that $\det(AB-BA)=0$

As pointed above by @Lord Shark the Unknown (whose comment struck me, pointing the right way) we have from Sylvester's inequality: $$0=\operatorname{rank}O_3=\operatorname{rank}(X^2) \geq \operatorname{rank}X+\operatorname{rank}X-3 \Rightarrow \operatorname{rank}X \leq 1.$$ Thus, $$\operatorname{rank}(XB-BX) \leq \operatorname{rank}(XB)+\operatorname{rank}(BX) \leq \operatorname{rank}X+\operatorname{rank}X \leq 2$$ and so $\det(AB-BA)=0$.

Here is a more or less direct, less creative solution. Since $(A-B)^2=0$, then either $A-B=0$ (in which case $AB-BA=0$), or its Jordan form is $$J=\begin{bmatrix} 0&1&0\\0&0&0\\0&0&0\end{bmatrix}.$$ So $A-B=SJS^{-1}$ for some $S$. Let $A'=S^{-1}AS$, $B'=S^{-1}BS$. Then $A'=B'+J$, and $$A'B'-B'A'=(B'+J)B'-B'(B'+J)=JB'-B'J.$$ Now check directly that $$ JB'-B'J=\begin{bmatrix}B'_{31}&B'_{32}&B'_{33}-B'_{11}\\ 0&0&-B'_{21}\\ 0&0&-B'_{31} \end{bmatrix}. $$ Thus $\det(JB'-B'J)=0$. Finally, \begin{align} \det(AB-BA)&=\det(SA'S^{-1}SB'S^{-1}-SB'S^{-1}SA'S^{-1})\\ \ \\ &=\det(A'B'-B'A')=\det(JB'-B'J)=0. \end{align}