An inequality on trace of product of two matrices

In the space of positive semi-definite matrices, trace is a proper inner-product (it is easy to show that), i.e. it obeys the Cauchy-Schwarz inequality: $\langle x,y \rangle \leq \sqrt{ \langle x,x \rangle \langle y,y\rangle}$. So

$$\mbox{tr}\{AB\}\leq \sqrt{\mbox{tr}\{A^2\} \mbox{tr}\{B^2\}}$$

Now, since $A$ is positive semidefinite, $\mbox{tr}\{A^2\} \leq \mbox{tr}\{A\}^2$, i.e., the eigenvalues of $A^2$ are squared eigenvalues of $A$, and since they are positive

$$\mbox{tr} \{A^2\} = \sum_{i=1}^{N}\lambda_{i}^{2}\leq \left( \sum_{i=1}^{N}\lambda_{i} \right)^{2} = \mbox{tr}\{A\}^2 \leq 1$$

A similar argument for B proves $\mbox{tr}\{B^2\}\leq 1$ . So $\mbox{tr}\{AB\}\leq 1$. Hope this answers your question.

First, note that $(A-B)^{2}$ is positive semi-definite, so we have:

$$0\leq\mathrm{Tr}(A-B)^{2}=\mathrm{Tr}(A^{2})+\mathrm{Tr}(B^{2})-2\mathrm{Tr}(AB)$$

$$\mathrm{Tr}(AB)\leq\frac{1}{2}(\mathrm{Tr}(A^{2})+\mathrm{Tr}(B^{2}))$$

Second, for $A$ positive semi-definite, suppose that all of eigenvalues are $\lambda_{1}$, $\lambda_{2}$, $\cdots$, $\lambda_{n}$, then $\lambda_{i}\geq0$ and $\mathrm{Tr}A=\sum_{i=1}^{n}\lambda_{i}\leq1$, so $\mathrm{Tr}(A^{2})=\sum_{i=1}^{n}\lambda_{i}^{2}\leq\sum_{i=1}^{n}\lambda_{i}\leq1$.

Similarly, $\mathrm{Tr}(B^{2})\leq1$, so $\mathrm{Tr}(AB)\leq1$.

Remark. More generally, we can conclude that the range of $\mathrm{Tr}(AB)$ is $[0,1]$.

As $A$ and $B$ are positive semi-definite, so there exist $C$ and $D$ such that $A=C^{T}C$ and $B=D^{T}D$, so $\mathrm{Tr}(AB)=\mathrm{Tr}(C^{T}CD^{T}D)=\mathrm{Tr}(CD^{T}DC^{T})=\mathrm{Tr}[CD^{T}(CD^{T})^{T}]\geq0$.

Set $A=diag[1,0,0,\cdots,0]$ and $A=diag[0,1,0,\cdots,0]$, then $\mathrm{Tr}(AB)=0$.

Set $A=B=diag[1,0,0,\cdots,0]$, then $\mathrm{Tr}(AB)=1$.

Accoading to above, we can conclude that $\text{Range}(\mathrm{Tr}(AB))=[0,1]$.

An extension to @dineshdileep 's answer can also be found here, in which it shows that: $$tr(A^TB)\le \sqrt{tr(A^TA)tr(B^TB)}$$