$\operatorname{Tr}(AB)=\operatorname{Tr}(A)\operatorname{Tr}(B)$
Here is a bit messy solution:
Lemma. Let $F$ be a field with characteristic $\neq 2$ and $A$ and $B$ be $2\times 2$ be matrices over $F$.
- If $A$ and $B$ commute, then either $A = pB + qI$ or $B = pA + qI$ for some $p, q \in F$.
- If $A^2 = k(AB - BA)$, then $A^2 = 0$. In particular, if $k \neq 0$, then $AB = BA$.
We postponed the proof of this lemma to the end. By rearranging the equality in the assumption, we obtain $(A+B)^2 = 2[(A+B)A-A(A+B)]$. So, by Lemma, $(A+B)^2 = 0$ and $AB = BA$. Now we claim the following.
Claim. Either $A = 0$ or $B = kA$ for some $k$.
Once this is true, then the rest is straightforward: If $A = 0$, there is nothing to prove. If $B = kA$ for some $k$, then
$$ \operatorname{Tr}(AB) = k\operatorname{Tr}(A^2) = k\operatorname{Tr}(A)^2 = \operatorname{Tr}(A)\operatorname{Tr}(B), $$
where the second step follows from the identity $\operatorname{Tr}(A^2) = \operatorname{Tr}(A)^2$, which itself is a consequence of both Cayley-Hamilton theorem and $\det(A) = 0$. $\square$
So it remains to show the claim. Assume that $A \neq 0$. Since $\det(A) = 0$, we may utilize Lemma to write
$$A+B = aA + bI $$
for some $a, b$. Indeed, this is obvious when $B = pA + qI$. On the other hand, if $A = pB + qI$, then $p \neq 0$ by the assumption, and so, $A+B = \frac{p+1}{p}A - \frac{q}{p}I$.
Next, we consider two possibilities:
If $a = 0$, then $0 = (A+B)^2 = b^2 I$, and so, $b = 0$. This implies $B = -A$.
If $a \neq 0$, then by rearranging $0 = (A+B)^2$ we obtain $A\left(A + \frac{2b}{a}I\right) = -\frac{b^2}{a^2} I$. Since $A$ is not invertible, this must imply $b = 0$ and hence $B = (a-1)A$.
This completes the proof of Claim. $\square$
Proof of Lemma. For the first part, replacing $A$ by $A-\frac{1}{2}\operatorname{Tr}(A)$ and $B$ by $B-\frac{1}{2}\operatorname{Tr}(B)$ if needed, we may assume that $A$ and $B$ take the form
$$ A = \begin{pmatrix} a & b \\ c & -a \end{pmatrix}, \qquad B = \begin{pmatrix} p & q \\ r & -p \end{pmatrix} $$
Now write $u = (a, p)$, $v = (b, q)$, $w = (c, r)$. Then comparing both sides of $AB - BA = 0$, we find that $AB = BA$ if and only if $\det(u, v) = \det(v, w) = \det(w, u) = 0$. In particular, they are parallel to each other. This implies that $A$ and $B$ are parallel, and so, the desired conclusion follows.
For the second part, we first check $\det(A) = 0$. To this end, assume otherwise. Then
$$ 2 = \operatorname{Tr}(I) = \operatorname{Tr}(A^{-1}A^2A^{-1}) = k \operatorname{Tr}(BA^{-1} - A^{-1}B) = 0,$$
a contradiction. Then taking trace to both sides, by Cayley-Halimton theorem,
$$\operatorname{Tr}(A)^2 = \operatorname{Tr}(A^2) = k \operatorname{Tr}(AB-BA) = 0, $$
and so, $\operatorname{Tr}(A) = 0$. Therefore the desired claim follows from Cayley-Hamilton theorem. $\square$
Sangchul Lee's answer can be slightly shortened as follows.
First of all, the key observation is that $$ (A+B)^2 = 2[B,A] = 2[A+B,A],\tag{1} $$ where the square bracket denotes a commutator.
By Jacobson's lemma, if $X$ is an $n\times n$ matrix over some field whose characteristic is either zero or $>n$, and $X$ commutes with some commutator $[X,Y]$, the commutator $[X,Y]$ must be nilpotent. So, in our case, if the characteristic of the underlying field is not $2$, since $A+B$ commutes with $(A+B)^2=2[A+B,A]$, Jacobson's lemma implies that $[A+B,A]$ is nilpotent. Hence $(A+B)^2$ and in turn $A+B$ are nilpotent and $(A+B)^2=0$. But then $(1)$ also implies that $[B,A]=0$, i.e. $A$ and $B$ commute.
Let $E$ be the algebraic closure of the field generated by the eight matrix entries in $A$ and $B$. Since $A$ and $B$ commute, they are simultaneously triangulable over $E$. This, together with the facts that $A$ is singular and $A+B$ is nilpotent, imply that $B$ must have a zero eigenvalue, i.e. $B$ is singular.
Let $a=\operatorname{tr}(A)$ and $b=\operatorname{tr}(B)$. On one hand, as both $A$ and $B$ are singular, by Cayley-Hamilton theorem we have \begin{align} \operatorname{tr}((A+B)^2) &=\operatorname{tr}(A^2+2AB+B^2)\\ &=\operatorname{tr}(aA+2AB+bB)\\ &=a^2+2\operatorname{tr}(AB)+b^2.\tag{2} \end{align} On the other hand, \begin{align} (\operatorname{tr}(A+B))^2 &=(\operatorname{tr}(A)+\operatorname{tr}(B))^2\\ &=a^2+2\operatorname{tr}(A)\operatorname{tr}(B)+b^2.\tag{3} \end{align} As $A+B$ is nilpotent, both $\operatorname{tr}((A+B)^2)$ and $(\operatorname{tr}(A+B))^2$ are zero. Now the result follows by equating $(2)$ and $(3)$.
Finally, note that the assumption that characteristic of the field is not $2$ is essential here. E.g. consider $$ A=\pmatrix{0&0\\ 1&0},\ B=\pmatrix{1&1\\ 0&1}. $$ If the ground field is $GF(2)$, the conditions that $A^2+3AB+B^2=BA$, or equivalently $(A+B)^2=0$ (because $1+1=0$), and that $\det(A)=0$ are both satisfied, but clearly $\operatorname{tr}(AB)=1\ne0=\operatorname{tr}(A)\operatorname{tr}(B)$ here.