Eigenvalues of $AB$ and $BA$ where $A$ and $B$ are square matrices

Here is a proof similar to what the OP has tried:

Let $\lambda$ be any eigenvalue of $AB$ with corresponding eigenvector $x$. Then

$$ABx = \lambda x \Rightarrow \\ BABx = B\lambda x \Rightarrow\\ BA(Bx) = \lambda (Bx) $$

which implies that $\lambda$ is an eigenvalue of $BA$ with a corresponding eigenvector $Bx$, provided $Bx$ is non-zero. If $Bx = 0$, then $ABx = 0$ implies that $\lambda = 0$.

Thus, $AB$ and $BA$ have the same non-zero eigenvalues.


It suffices to show that $AB$ and $BA$ have the same characteristic polynomial. First assume that $A$ is invertible then

$$\chi_{AB}(x)=\det(AB-xI)=\det A\det(B-xA^{-1})\\=\det(B-xA^{-1})\det A=\det(BA-xI)=\chi_{BA}(x)$$ Now since $\operatorname{GL}_n(K)$ is dense in $\operatorname{M}_n(K)$ then there's a sequence of invertible matrices $(A_n)$ convergent to $A$ and by the continuity of the $\det$ function we have $$\chi_{AB}(x)=\det(AB-xI)=\lim_{n\to\infty}\det(A_nB-xI)=\lim_{n\to\infty}\det(BA_n-xI)\\=\det(BA-xI)=\chi_{BA}(x).$$


Alternative proof #1:

If $n\times n$ matrices $X$ and $Y$ are such that $\mathrm{tr}(X^k)=\mathrm{tr}(Y^k)$ for $k=1,\ldots,n$, then $X$ and $Y$ have the same eigenvalues.

See, e.g., this question.

Using $\mathrm{tr}(UV)=\mathrm{tr}(VU)$, it is easy to see that $$ \mathrm{tr}[(AB)^k]=\mathrm{tr}(\underbrace{ABAB\cdots AB}_{\text{$k$-times}}) =\mathrm{tr}(\underbrace{BABA\cdots BA}_{\text{$k$-times}})=\mathrm{tr}[(BA)^k]. $$ Now use the above with $X=AB$ and $Y=BA$.

Alternative proof #2:

$$ \begin{bmatrix} I & A \\ 0 & I \end{bmatrix}^{-1} \color{red}{\begin{bmatrix} AB & 0 \\ B & 0 \end{bmatrix}} \begin{bmatrix} I & A \\ 0 & I \end{bmatrix} = \color{blue}{\begin{bmatrix} 0 & 0 \\ B & BA \end{bmatrix}}. $$ Since the $\color{blue}{\text{red matrix}}$ and the $\color{red}{\text{blue matrix}}$ are similar, they have the same eigenvalues. Since both are block triangular, their eigenvalues are the eigenvalues of the diagonal blocks.