Why is a symmetric traceless tensor zero when averaged over all directions?
The order parameter is a meso- or macroscopic quantity which is formed by adding together the tensor contributions from a bunch of different molecules: if each molecule $\alpha$ has a tensor $T^{(\alpha)}$, then the total order parameter is the sum of the $T^{(\alpha)}$ from all the molecules in the relevant sample of material, i.e., $$ T = \sum_\alpha T^{(\alpha)}, $$ or, when seen as a bilinear function with explicit arguments, $$ T(u,v) = \sum_\alpha T^{(\alpha)}(u,v), $$ for arbitrary $u,v\in\mathbb R^3$.
The book's claim is that they want $T$ to be zero in the disordered state, in which the $T^{(\alpha)}$ themselves are pointing in every direction: that is,
- you fix some arbitrary reference tensor $T^{(0)}$,
- you assert that each of the $T^{(\alpha)}$s is a rotated version $R[T^{(0)}]$ of the reference tensor,
- and you assert that the ensemble combination $\sum_\alpha$ samples over all possible orientation rotations $R$, i.e., $$ T(u,v) = \int_\mathrm{SO(3)} R[T^{(0)}](u,v) \:\mathrm dR. $$
Here the integral over $\mathrm{SO}(3)$ must be taken over the Haar measure of the group, which is the only definition of integration which is invariant w.r.t. the group action.
You might also be wondering what the rotated tensor "$R[T^{(0)}](u,v)$" means. In the abstract, it means that you rotate the tensor, but in practice, you can flip that "active transformation" view on its head into a "passive" one, by simply rotating the whole system ($R[T^{(0)}]$, $u$ and $v$) back to the original orientation, which gives you $$R[T^{(0)}](u,v) = T^{(0)}(R^{-1}u,R^{-1}v)$$ as a concrete recipe for evaluating $R[T^{(0)}]$.
The central claim here is the following:
Claim: this orientation averaging reduces the tensor to a multiple of the isotropic tensor $\delta(u,v) = u\cdot v$, with components proportional to the Kronecker delta $\delta(e_i,e_j) = \delta_{ij}$. Moreover, the coefficient is proportional to the trace of the reference tensor:
\begin{align} T(u,v) & = \int_\mathrm{SO(3)} R[T^{(0)}](u,v) \:\mathrm dR \\ & = \frac13 \mathrm{tr}(T^{(0)}) \delta(u,v) . \end{align}
(As pointed out by Valter Moretti, this is basically a whittled-down version of Schur's lemma, a central result in group representation theory which is well worth learning about if you feel like digging deeper.)
There are two main ways to understand this result: there is a fundamental core intuition that makes it work, but you can also prove it rigorously.
Intuition
The core of the result is that the orientation averaging means that $T(u,v)$ must be an isotropic tensor, i.e., it must be such that $$ T(Ru,Rv) = T(u,v) $$ for every $R\in\mathrm{SO}(3)$. This is an extremely strong constraint: up to a constant multiple, there is only one isotropic tensor of rank two $-$ the Kronecker delta. This implies that $T(u,v) = \lambda \delta(u,v)$, and you just need to determine $\lambda$. This is extremely simple to do, by just taking the trace: $$ \mathrm{tr}(T) = \lambda \: \mathrm{tr}(\delta) = \lambda \sum_{i=1}^3 \delta(e_i,e_i) = 3\lambda, $$ so $\lambda = \frac 13 \mathrm{tr}(T)$.
Proof
To complete the rigorous proof, thus, we only need to show that the Kronecker delta is the only isotropic tensor. As in V. Moretti's answer, the simplest way to do this is to consider the action of $T$ on a basis $\{ e_i\}$.
- For the off-diagonal elements, $T(e_i,e_j)$ with $i\neq j$, there exists a rotation $R$ (specifically, a rotation by $180°$ about $e_i$) for which $Re_i=e_i$ and $Re_j=-e_j$, which means that $$ T(e_i,e_j) = T(Re_i,Re_j) = T(e_i,-e_j) = -T(e_i,e_j), $$ and therefore all the off-diagonal elements must vanish.
- With this in hand, we only need to show that all the diagonal elements are equal. Choosing $e_i$ and $e_j$ with $i\neq j$, a rotation by $\pm 90°$ about $e_k$ does $Re_i=e_j$, so $$ T(e_i,e_i) = T(Re_i,Re_i) = T(e_j,e_j). $$
This completes the proof: $T(u,v)$ and $\lambda\:\delta(u,v) = T(e_1,e_1)\delta(u,v)$ coincide on a basis, so they must coincide everywhere.
OK, another proof according to the Remark by Emilio that made clear the interpretation of the question. We have to average the tensor (as a bilinear map) in the space of the rotations. The only way to do it, as Emilio wrote, is to take an integral with respect to the normalized (left-invariant) Haar measure of $SO(3)$: $$\langle T\rangle_{SO(3)}(u,v) := \int_{SO(3)} T(R^tu,R^tv) dR$$ Since $dB^tR = dR$ for every $B\in SO(3)$, $$\langle T\rangle_{SO(3)}(Bu,Bv) = \int_{SO(3)} T(R^tBu,R^tBv) dR = \int_{SO(3)} T((B^tR)^tu,(B^tR)^tv) dR $$ $$ = \int_{SO(3)} T((B^tR)^tu,(B^tR)^tv) dB^{t}R = \int_{SO(3)} T(R'^tu,R'^tv) dR' = \langle T\rangle_{SO(3)}(u,v)\:.$$ In matricial form, writing $R$ for $B$, $$u^tR \langle T\rangle_{SO(3)} R^tv = u^t\langle T\rangle_{SO(3)}v\:. $$ Since $u,v$ are arbitrary, $$R \langle T\rangle_{SO(3)} R^t = \langle T\rangle_{SO(3)}\:. $$ That is $$R \langle T\rangle_{SO(3)}= \langle T\rangle_{SO(3)}R \quad \forall R \in SO(3)\:.$$ Since the fundamental representation of $SO(3)$ is irreducible, Schur's lemma implies that, for some constant, $t\in \mathbb{R}$ $$ \langle T\rangle_{SO(3)} = tI\:.$$ In particular, since every unit vector can be transformed to any other unit vector with a rotation, an orthonormal a basis is transformed into an orthonormal basis, and the trace does not depend on the basis, $$3t = \sum_{i=1}^3\langle T\rangle_{SO(3)}(e_i,e_i)$$ $$ = tr\langle T\rangle_{SO(3)} = \int_{SO(3)} \sum_{i=1}^3T(R^te_i,R^te_i) dR = \int_{SO(3)} tr(T) dR = tr(T)\:.$$ In summary $$ (\langle T\rangle_{SO(3)})_{ab} = \frac{1}{3}tr(T)\delta_{ab}\:.$$
Symmetry requirement is not necessary. Let us take an orthonormal basis $e_1,e_2,e_3$ and consider all the unit vectors $n \in S^2$. $$\int_{S^2} T(n,n) d n = \sum_{i,j=1}^3T(e_i,e_j) \int_{S^2} n^i n^j dn\:.$$ $dn$ is the standard rotation-invariant measure on $S^2$ with total value $4\pi$, i.e., referring to standard spherical coordinates $$dn = \sin \theta d\theta d\phi\:.$$ Now, for every choice of $i=1,2,3$ and $j=1,2,3$ with $i\neq j$ there is a rotation $R$ such that $(Rn)^i=-n^i$ but $(Rn)^j = n^j$. Since $dn = dRn$ we immediately have that $$\int_{S^2} n^i n^j dn= -\int_{S^2} n^i n^j dn= 0 $$ if $i\neq j$. Therefore \begin{align} \sum_{i,j=1}^3T(e_i,e_j) \int_{S^2} n^i n^i dn & = \sum_{i=1}^3T(e_i,e_i) \int_{S^2} n^i n^i dn \\ & = T(e_1,e_1)\int_{S^2} (n^1)^2 dn + T(e_2,e_2)\int_{S^2} (n^2)^2 dn \\ & \qquad \quad + T(e_3,e_3)\int_{S^2} (n^3)^2 dn\:. \end{align} To conclude, observe that rotational invariance (as above) $$\int_{S^2} n^i n^i dn = d \quad \mbox{independently of $i=1,2,3$}$$ and thus $$4\pi = \int_{S^2} \sum_{i=1}^3 (n^i)^2 dn = 3d$$ which implies $$d = \frac{4\pi}{3}\:.$$ Inserting in the formula above \begin{align} \int_{S^2} T(n,n) d n & = \sum_{i,j=1}^3T(e_i,e_j) \int_{S^2} n^i n^j dn \\ \frac{4\pi}{3} \sum_{i=1}^3 T(e_i,e_i) & = \frac{4\pi}{3} \mathrm{tr}(T)\:. \end{align} In other words, averaging over all directions we obtain the trace up to the factor $1/3$: $$\langle T\rangle_{S^2} := \frac{1}{4\pi}\int_{S^2} T(n,n) d n = \frac{1}{3} \mathrm{tr}(T)\:.$$ Your hypothesis also requires $\mathrm{tr}(T)=0$ concluding the proof.