How prove this two symmetric matrices $AB=0$

This answer is an amalgamation of ideas from two deleted posts by @julien and @user1551 together with some of my thoughts.

Following @julien's insight, let us look at the polynomial condition at $n = 4$. If one compare the coefficient of the $x^2 y^2$ terms on both sides, we get

$$2\text{tr}(A^2B^2) + \text{tr}( (AB)^2 ) = 0$$

Combine this with the fact

$$\text{tr}((AB+BA)^2) = \text{tr}((AB+BA)^T(AB+BA)) \ge 0 \\\implies \text{tr}(A^2B^2) + \text{tr}( (AB)^2 ) \ge 0$$

We get $\text{tr}(A^2B^2) \le 0$. However

$$\text{tr}(A^2B^2) = \text{tr}(ABBA) = \text{tr}((BA)^\top BA ) \ge 0$$

This implies $\text{tr}(A^2B^2) = 0$.

Following @user1551's idea. Since $A$ is real symmetric, we can choose a basis such that $A$ is a diagonal matrix and its first $k=\text{rank}(A)$ diagonal entries are the only non-zero entries. $\text{tr}(A^2B^2)$ then become a positive linear combination of the first $k$ diagonal entries of $B^2 = B^\top B$ which are non-negative themselves.

As a result, $\text{tr}(A^2 B^2) = 0$ implies the first k diagonal entries of $B^2$ vanish. This means the first $k$ rows/columns of $B$ are zero and hence $AB=0$.

Update

It turns out there is a much simpler argument. The expression $ \displaystyle \| C \|_F = \sqrt{ \text{tr}( C^\top C ) }$ is the famous Frobenius norm over the ring of real matrices! The vanishing of the coefficient of the $x^2 y^2$ terms $$\text{tr}((AB+BA)^2) + 2\text{tr}(BAAB) = 0$$ is equivalent to $$\quad \|AB+BA\|_F^2 + 2\|AB\|_F^2 = 0$$ This implies $\|AB\|_F = 0$ and hence $AB = 0$.