orthogonal complement of symmetric matrices
Notice that $\langle A, B \rangle = \sum_{i,j} [A]_{ij} [B]_{ij}$. Let $E_{ij} = e_i e_j^T$, ie, the zero matrix except for a one in the $(i,j)$ position.
Suppose $\langle A, S \rangle = 0$ for all symmetric matrices, then it is true for $S=E_{ij}+E_{ji}$. This gives $\langle A, E_{ij} \rangle + \langle A, E_{ji} \rangle = 0$, which gives $[A]_{ij} + [A]_{ji} = 0$, from which it follows that $A = -A^T$.
Now suppose that $A$ is skew symmetric, and $S$ is symmetric. Then we can write $S = U^T + \Lambda + U$, where $\Lambda$ is diagonal, and $U$ is strictly upper triangular. Then $\langle A, \Lambda \rangle = 0$ because $[A]_{ii} = 0$. Since $\operatorname{tr} M = \operatorname{tr} M^T$ we also have $\langle A, U^T \rangle = \langle A^T, U \rangle$, and skew symmetry gives $\langle A^T, U \rangle = - \langle A, U \rangle$. Hence $\langle A, S \rangle = \langle A, U^T \rangle + \langle A, \Lambda \rangle - \langle A, U^T \rangle = 0$.
Hints: Presumably you are talking about real square matrices. You have to show that:
- Every real square matrix $A$ can be written as the sum of a symmetric matrix $H$ and a skew symmetric matrix $K$, i.e. $A=H+K$. Suppose $A=H+K$. Note that $H$ is symmetric means $H^T=H$ and $K$ is skew symmetric means $K^T=-K$. Can you express $A^T$ in terms of $H$ and $K$? Now try to back out $H$ and $K$ from $A$ and $A^T$.
- In step 1, you have shown that the real matrix space is the sum of the subspace of symmetric matrices and the subspace of skew symmetric matrices. It remains to show that the latter two subspaces are orthogonal to each other. That is, you have to show that $\langle H,K\rangle=\operatorname{tr}(H^TK)=0$ for every symmetric $H$ and every skew symmetric $K$. Note that $H^T=H$, so it boils down to proving that $\operatorname{tr}(HK)=0$. The trace identities $\operatorname{tr}(M)\equiv\operatorname{tr}(M^T)$ and $\operatorname{tr}(XY)\equiv\operatorname{tr}(YX)$ are useful.
It is enough to show $\langle A,B \rangle=Tr(A^\top B)=0$ for $A\in Sym(n),~B\in Skew(n)$.
Indeed,
$$Tr(A^\top B) \underset{(1)}{=} Tr(B^\top A)\underset{(2)}{=}Tr(-B A) \underset{(3)}{=} -Tr(AB)\underset{(4)}{=} -Tr(A^\top B)$$
(1): $\langle A ,B\rangle = \langle B ,A\rangle$
(2): $B\in Skew(n)$
(3): Cyclic property of trace
(4): $A\in Sym(n)$
Thus, we concludes that $Tr(A^\top B)=0$