Why is the $S_{z} =0$ state forbidden for photons?

Massless particles with spin do not have a "$S_z = 0$" state because they actually do not have spin like massive particles do. They have helicity, which is the value of the projection of the spin operator onto the momentum operator. The reason for this is the representation theory of the group of spacetime symmetry, the Poincaré group.

To understand this, we must first recall that "spin" is the number that labels irreducible representations of $\mathrm{SU}(2)$, the double cover of the rotation group $\mathrm{SO}(3)$. But, in relativistic quantum field theory, which is the theory needed to describe photons, this rotation group is not the spacetime symmetry group we need to represent. Instead, we must seek representations of the identity connected component of the Poincaré group $\mathrm{SO}(1,3)\rtimes\mathbb{R}^4$, i.e. of the proper orthochronous Lorentz transformation together with translations.

Now, for the finite-dimensional representations of the Lorentz group, we're lucky in that there is an "accidental" equivalence of algebra representations of $\mathfrak{so}(1,3)$ and $\mathfrak{su}(2)\times\mathfrak{su}(2)$, allowing us to label the finite-dimensional representations in which classical relativistic fields transform by pairs of half-integers $(s_1,s_2)$ where $s_i\in\frac{1}{2}\mathbb{Z}$ labels a single $\mathfrak{su}(2)$ representation. The actual rotation algebra sits diagonally in this $\mathfrak{su}(2)\times\mathfrak{su}(2)$, so the physical spin of such a representation is $s_1+s_2$. This determines the classical spin associated to a field.

As so often, the quantum theory makes things more complicated: Wigner's theorem implies that we must now seek unitary representations of the Poincaré group on our Hilbert space of states. Except for the trivial representation corresponding to the vacuum, none of the finite-dimensional representations is unitary (essentially because the Poincaré group is non-compact and doesn't have compact normal subgroups). So we must turn to infinite-dimensional representations, and here we do not have the equivalence between $\mathfrak{so}(1,3)$ and $\mathfrak{su}(2)\times\mathfrak{su}(2)$. The techniques exploited to realize this equivalence explicitly rely on finite-dimensionality of the representation. In particular, there is no such isomorphism as $\mathrm{SO}(1,3)\cong\mathrm{SU}(2)\times\mathrm{SU}(2)$, regardless of how often you will read similar claims in physics books. For more on this issue, see e.g. this answer by Qmechanic.

It turns out that classifying the unitary representations is not so simple a task. The full classification is called Wigner's classification, and it turns out that to construct irreducible unitary representations, it is relevant to look at the little group corresponding to the momentum of a particle - the subgroup of the Lorentz group which leaves the momentum of the particle invariant. For a massive particle, this is $\mathrm{SO}(3)$, and it turns out we can label the unitary representation also with our familiar spin $s$.

But for a massless particle, the momentum $(p,-p,0,0)$ is not invariant under $\mathrm{SO}(3)$, but under a group called $\mathrm{ISO}(2)$ or $\mathrm{SE}(2)$, which is essentially $\mathrm{SO}(2)$ with translations. Being Abelian, $\mathrm{SO}(2)$ has only one-dimensional irreducible representations, labeled by a single number $h$, which turns physically out to be the eigenvalue of helicity. There are more general cases for $\mathrm{ISO}(2)$, called the continuous spin representations (CSR), but they have so far not been physically relevant.

Now, this single number $h$ flips its sign under parity, so for particles associated to classical fields with non-zero spin, we must take both the $h$ and the $-h$ representations. And that's it - massless particles of helicity $h$ have the $h\oplus -h$ representation on their space of states, not a spin representation of $\mathrm{SO}(3)$. Evaluation of the actual spin operator shows that the our classical idea of spin coincides with the number $h$.

Therefore, without having said anything about the photon or the electromagnetic field in particular, we know that massless particles of non-zero spin come with two degrees of freedom. This is completely general, and at the heart of the argument that all massless vector bosons are gauge bosons:

We know that a generic vector field has three d.o.f. - the independent field components that transform into each other under Lorentz transformation, hence three independent sets of creation and annihiliation operators that transform into each other, hence we expect three distinct kinds of particle states.

But the two d.o.f. of a massless spin-1 particle don't match with this - so one of the d.o.f. of a massless vector field must be "fake". The way d.o.f.s of fields are "fake" is by the field being a gauge field and there being 1 d.o.f. in the freedom to choose a gauge. The story of the quantization of gauge theory - even in the Abelian case of electromagnetism - is subtle, and you are right to not blindly accept the argument that the two classical polarizations of the gauge field - the longitudinal one is eliminated by gauge symmetry - become distinct kinds of particle states in the quantum theory: The decoupling of the states one would naively associate to the longitudinal modes is ensured by the Ward identities, and not at all obvious a priori.

It is by this that the properties of being a gauge boson and of not having a $S_z = 0$ and of being massless are all interrelated: Being one of these things immediately also forces the other two. In this answer, I regarded "being massless" as the fundamental property, since this shows "no $S_z=0$" without assuming anything more specific about the field - in particular, without restricting to gauge fields or electromagnetism a priori.


I can't improve on KDN's answer, but given Todd's comments this is an attempt to rephrase KDN's answer in layman's terms.

A system is only in an eigenstate of spin around an axis if a rotation about the axis doesn't change the system. Take $z$ to be the direction of travel, then for a spin 1 system the $S_z$ = 0 state would be symmetric to a rotation about an axis normal to the direction of travel. But this can only be the case if the momentum is zero i.e. in the rest frame. If the system has a non-zero momentum any rotation will change the direction of the momentum so it won't leave the system unchanged.

For a massive particle we can always find a rest frame, but for a massless particle there is no rest frame and therefore it is impossible to find a spin eigenfunction about any axis other than along the direction of travel. This applies to all massless particles e.g. gravitons also have only two spin states.


The answers from KDN and John Rennie are right - I'll just try to illustrate how it works:

The components of a massless spin 1 field satisfy $$\Box^2 A_{\mu}(x) = 0$$ Traditionally we perform the expansion in momentum variables $$ A^{\mu}(x) = \int{\frac{1}{\sqrt{p^0}}A^{\mu}({\bf{p}})e^{-ip.x}}d^3{\bf{p}} + \textrm{c.c.}$$ If the particle is moving in the z direction, then its momentum is $$ p^{\mu} = (p^0, 0, 0, p^3)$$ and the Lorenz condition $\partial_{\mu}A^{\mu}=0$ which, on the momentum space variables looks like $$p_{\mu}A^{\mu}({\bf{p}})=0 $$ now becomes $$ p^0A^{0}({\bf{p}})-p^3A^{0}({\bf{p}})=0$$ and so we see that $$ A^{0}({\bf{p}}) = A^{3}({\bf{p}})$$ So we can express $A^{\mu}({\bf{p}})$ in terms of polarization vectors $$ A^{\mu}({\bf{p}}) = \sum\limits_{\lambda}a_{\lambda}({\bf{p}})\epsilon^{\mu}_{\lambda}$$ where the three polarization vectors look like $$ \epsilon^{\mu}_{1}=(0, 1, 0, 0)$$ $$ \epsilon^{\mu}_2=(0, 0, 1, 0)$$ $$ \epsilon^{\mu}_{3}=(1, 0, 0, 1)$$ If you now take the special case of a wave with just the third polarization $$ A^{\mu}(x) = \int{\frac{1}{\sqrt{p^0}}a_{3}({\bf{p}})\epsilon_3^{\mu}e^{-ip.x}}d^3{\bf{p}} + \textrm{c.c.}$$ and you now compute the ${\bf{E}}$ and ${\bf{B}}$ fields, then the special form of $\epsilon_3^{\mu}$ ensures you get zero. Hence the polarization in the direction of propagation does nothing to contribute to the field.