Building the meson octet and singlet

1. The Lie group $G$ behind the meson octet is the $SU(3)_{\rm Flavor}$ Lie group over the three lightest quark flavors $u$, $d$, and $s$. More precisely, the fundamental $SU(3)$ representation $$V~=~\text{Fund} ~=~{\bf 3}$$ is a linear span of the $u$, $d$, and $s$ quarks. The three complex coefficients are collected in a $3\times 1$ column vector $v\in V$.

2. How does the $G=SU(3)_{\rm Flavor}$ act on $V$? It acts from left by multiplying $v\in V$ with a $3\times3$ special unitary group matrix $g\in G$, which results in a new column vector $v'=gv\in V$. Similarly, $G$ acts on the complex conjugated representation $\bar{V}$ as $\bar{v}'=\bar{g}\,\bar{v}$. In particular, if we write $\bar{v}$ as a $1\times 3$ row vector $v^{\dagger}$, the group acts as $v^{\dagger \prime}=v^{\dagger}g^{\dagger}=v^{\dagger}g^{-1}$.

3. Lie group representation theory explains how the meson nonet decomposes in irreps, $$\begin{align} {\bf 3} \otimes \overline{\bf 3}~=~&V \otimes \bar{V}\cr ~=~&\text{Fund} \otimes \overline{\text{Fund}}\cr ~\cong~& \text{Adj}\oplus \text{Sing}\cr ~=~& {\bf 8}\oplus {\bf 1}. \end{align}$$

4. We may identify $$\begin{align}V \otimes \bar{V}~\cong~&V \otimes V^\dagger\cr ~\cong~&{\rm Mat}_{3 \times 3}(\mathbb{C}).\end{align}\tag{*}$$ Notice that $G$ acts on both spaces $V \otimes V^\dagger$ and ${\rm Mat}_{3 \times 3}(\mathbb{C})$ by similarity transformations.

5. The $SU(3)$ singlet is the eta prime meson $$\eta'~=~\frac{u\otimes \bar{u}+ d\otimes \bar{d}+ s\otimes \bar{s}}{\sqrt{3}}.$$ Under the identification (*), the $\eta'$ corresponds to a $3 \times 3$ matrix proportional to the ${\bf1}_{3 \times 3}$ unit matrix. How do we know this is the unique flavor singlet? On one hand, the singlet is characterized by being invariant under the group action, i.e. similarity transformations. On the other hand, we know from Schur Lemma, that the only matrices that are invariant under all similarity transformations are the ones proportional to the unit matrix. Equivalently, in terms of the corresponding Lie algebra $su(3)$, the only matrices that commute with all the Lie algebra generators are the ones proportional to the unit matrix. This latter condition may be viewed as the constraint that OP is requesting.

6. Finally let us mention, that the $SU(3)_{\rm Flavor}$ symmetry is only an approximate symmetry in the standard model, as is evident from the different meson masses. The $SU(3)_{\rm Flavor}$ symmetry can be decomposed via branching rules in (strong) $SU(2)$ isospin symmetry, see e.g. chapter 10 of 't Hooft's lecture notes. The pdf file is available here.

The constraint on the singlet state $\eta' = \frac{1}{\sqrt{3}}(u\bar u+d\bar d+s\bar s)$ is that it be flavorless, just as the constraint on the singlet state for SU(2) is that it have no angular momentum.

To decompose $3\times 3^* = 8 + 1$ you first need to know what are the irreps. You can build these with raising and lowering operators as is done with SU(2).

If you are familiar with the graphical way of building and decomposing reps for SU(2) you will find there is an analogous method for SU(3). Basically, the reps look like triangles and hexagons and there is a way to multiply them graphically. (Of course, you can decompose them with raising and lowering operators as well, but you will find this tedious for SU(3).) This method allows you to calculate with pictures and very quickly that, for example, $3\times 3^* = 8 + 1$ and $3^*\times 3^* = 6^* + 3$. Of course, at some point you will probably want to learn Young-tableau.

I will avoid writing further details and drawing pictures and instead provide you with a reference that will make this all clear at the level you are looking for. See Ch 4, section 2.

Ta-Pei Cheng and Ling-Fong Li. Gauge Theory of Elementary Particle Physics