Angular Momentum: How do we know that $\hat{J}_z$ have non-degenerate eigenvalues?

In fact, we do not know!

The point is that in almost all textbooks two facts are confused: one is a physical fact the other is a mathematical fact.

Let us start from physics, by assuming that a physical system $S$, which is described in the Hilbert space $H_S$, admits a (strongly continuous) unitary representation of $SU(2)$ (or projective of $SO(3)$) representing the action of the physical rotations with respect to an inertial reference frame and an origin in the rest space of it: $$SU(2) \ni r \mapsto U(r) = e^{-\frac{i}{\hbar} \sum_{k=1}^3 \theta_k J_k}\: \quad \left(\mbox{where}\quad r = e^{-\frac{i}{2} \sum_{k=1}^3 \theta_k \sigma_k}\right)$$ By definition the three selfajoint generators $J_1,J_2,J_3$ of that representation are the observables total angular momentum of the physical system.

Now we pass to mathematics. The famous Peter-Weyl theorem establishes that every strongly continuous unitary representation of a compact group $G$ (as $SU(2)$) can be decomposed as a direct orthogonal sum of finite-dimensional irreducible representations of the said topological group. Hence, up to unitary isomorphisms, $$H = H^{(j_1)} \oplus H^{(j_2)} \oplus \cdots\tag{1}$$ where the set of parameters $j_k$ labels the irreducible representation $$G \ni g \mapsto U^{(j)}(g) : H^{(j)} \to H^{(j)}$$ and it can happen that $j_k=j_h$ (giving rise to degeneration as we discuss shortly).

In the case of $G=SU(2)$, the irreducible representations are labelled by $j =0, 1/2, 1, \ldots$. Each space $H^{(j)}$ is spanned by the familar basis $$|j, m\rangle$$ where $$m= -j, -j+1, \ldots, j-1, j$$ so that $\dim H^{(j)} = 2j+1$.

Here, by construction, there is no degeneration in each space $H^{(j)}$: once you fix $m$ therein you have completely fixed the vector $|j, m\rangle$.

The popular procedure that uses $J_\pm$ is actually exploited to construct the irreducible representations $H^{(j)}$ of $SU(2)$ and irreducibility is, in this case, equivalent to non-degenerateness. But this is a pure mathematical fact, it does not mean that the angular momentum is non-degenerate, since it is defined in $H$ which contains many copies of the spaces $H^{(j)}$. $$J_k = \oplus J_k^{(j_m)}\quad k=1,2,3\:.$$

The fact that a vector is completely fixed when choosing the eigenvalue of $J^2$ and $J_z$ is decided by physics analysing the above decomposition (1) in every concrete case as I am going to show.

Let us come back to physics to see how the generation may take place. In view of the said theorem, our Hilbert space $H_S$ will be decomposed as $$H_S = \oplus_{k \in K} H^{(j_k)}$$ where $j_k \in \{0, 1/2, 1, \ldots\}$ and, depending on the nature of the physical system, some values can be repeated. Suppose that, in a certain case, we find $$H_S = H^{(j_1)} \oplus H^{(j_2)}$$ and $j_1=j_2 =1$. In this case $H^{(1)}$ occurs twice and we can describe $H^{(1)}\oplus H^{(1)}$ using a basis $|1,m,l\rangle$ where $m=-1,0,1$ and $l=1,2$ with $$|1, m, 1> = |1,m> \oplus \: 0\quad \mbox{and}\quad \quad |1, m, 2> = 0 \oplus |1,m>\:.$$ Since $J_k = J^{(1)}_k\oplus J^{(1)}_k$, here your type of degenearation pops out: both $|1, m, 1>$ and $|1, m, 2>$ are eigenstates of $J^2$ and $J_z$ with the same eigenvalues $j(j+1)=2$ and $m$.

Mathematically speaking, degeneration is always due to the occurrence of many irreducible representations of the same type and this is decided by physics in the concrete case. It strictly depends on the nature of the studied physical system.

Let us consider a popular example of physical relevance. For a (spin $0$) particle in $\mathbb{R}^3$ the previous theory applies. In this case the Hilbert space is the tensor product $$H_S = L^2(\mathbb{R}^3) \simeq L^2(S^2)\otimes L^2([0,+\infty), r^2dr)$$ The first factor in the right most side describes the angular degrees of freedom, whereas the second one describes the radial degree of freedom. The first factor can be decomposed as $$L^2(S^2) = \oplus_{j=0,1,2, \ldots}H^{(j)}$$ where in terms of spherical harmonics $$H^{(j)} \ni |j,m\rangle = Y^j_m$$ Here no repetitions arise since the values of $j$ are all different. However repetitions show up when taking the second factor $L^2([0,+\infty), r^2dr)$ into account. Fix a Hilbert basis $\{y_n\}_{n=0,1,\ldots} \subset L^2([0,+\infty), dr)$ with some physical meaning (it does not matter now). We have $$L^2([0,+\infty),r^2 dr) = \oplus_{k=0,1,\ldots} K_n\:,$$ where $$K_n := span(y_n)\:.$$ In summary, $$H_S = (\oplus_{j=0,1,2, \ldots}H^{(j)}) \otimes (\oplus_{k=0,1,\ldots} K_n)$$ $$=\oplus_{j, n}(H^{(j)}\otimes K_n) $$ every space $H^{(j)}\otimes K_n$ is isomorphic to $H^{(j)}$ since $K_n$ is one dimensional.

The decomposition $$H_S =\oplus_{j, n}(H^{(j)}\otimes K_n) $$ is therefore another way to write (1) specialised to this case.

In this case, we have infinite degeneration: for every fixed $j,m$ we have an infinite number of vectors $$|j,m, n\rangle := |j, m\rangle |y_n\rangle$$ (one for each representation $H^{(j)}\otimes K_n \simeq H^{(j)}$). Every vector $|j,m, n\rangle$ is eigenvector of $J^2$ and $J_z$ with the same eigenvalues $j(j+1)$ and $m$, but we can also fix $n$ arbitrarily in $\mathbb{N}$.

Well, just like the principal ($n$) and azimuthal quantum numbers ($l$), the magnetic quantum number ($m$) is also an index used for referencing the wave functions. The degeneracy of fermionic states depends on several factors, which is out of scope for this discussion.

But eigenkets of $m$ are $\textbf{indeed degenerate}$. For each permissible value of $m$, we have $2S+1$ quantum states, indexed by their spin quantum number $S$. For electrons, we have ($s=\pm \frac{1}{2}$) and the states are given by $\lvert \frac{1}{2} \rangle$ and $\lvert -\frac{1}{2} \rangle$.

The degree of degeneracy of a quantum particle depends on the internal degrees of freedom of the particle. Till now, experimental conditions known have exacted only $4$ quantum numbers, which is a correct estimation of the quantum numbers, as proven by particles following Pauli's Exclusion Principle.

Hope this helps.