What experiment confirms $\mathbf{J}^2 = \hbar^2 j(j+1)$?

While it's not in the context of an atom, molecular rotations provide a particularly "clean" demonstration of the fact that $\mathbf{J}^2 = \hbar^2 j (j+1)$. In particular, it's not hard to show that the rotational KE of two point masses connected by a rigid rod is $$ H = \frac{J^2}{2 I} \quad \Rightarrow \quad E_j= \frac{\hbar^2 j(j+1)}{2 I}, $$ where $I$ is the moment of inertia about the center of mass. In particular, this means that transition energies between adjacent rotational energy states are $$ \Delta E = E_{j} - E_{j-1} = \frac{\hbar^2}{2I} j [j + 1 - (j-1)] = \frac{\hbar^2 j}{I}, $$ i.e., they are evenly spaced by energies of $\hbar^2/I$.

This even spacing has been confirmed by many experiments, and the values of $I$ inferred from these spectra are consistent with other measures of the molecular dimensions. Below is shown the absorption spectrum of carbon monoxide; the evenly spaced absorption lines are evident. While the large absorption lines of the molecules composed of carbon-12 and oxygen-16 (the most common isotopes of carbon and oxygen) are the most obvious features, we can also distinguish smaller absorption lines from molecules composed of carbon-12 and oxygen-18. Molecules composed of different isotopes will have the same bond lengths but different moments of inertia due to the the different masses of their constituent atoms; this results in a different spacing of the rotational lines.

^{From Hollas, Modern Spectrosopy, 1992; taken from these lecture notes.}

The Zeeman effect, discovered in the 19th century, was understood in quantum theory by Lande and Sommerfeld early on. Essentially, in a magnetic field, the energy levels of atoms are split by differences of magnetic moments, which are proportional to $J_z$, or the corresponding S and L components comprising J. This then allows to count the degenerate states so split of a system with fixed J (S, L, etc...) in a mixed population, graphically and dramatically. So at the most primitive level, the effect is a handle on degeneracy, $2j+1$.

This degeneracy, $2j+1$, is an easy counting observable, emerging as an eigenvalue of the formal operator $$ \sqrt{4\mathbf{J}^2/\hbar^2+1} , $$ the ready hands-on manifestation of the quantum number j, which bounds the eigenvalues' magnitude $|j_z|$ from above. Even if you had no idea about quantization, the discreteness of these eigenvalues would cue you in to the quantum nature of the lines' spacings, and the integer nature of all these quantities. In the 1920s, Wigner and his cohorts changed the face of physics fully understanding these regularities in terms of the theory of the rotation group (Their injection of theory was termed "Gruppenpest" --Group pestilence--by the experimental community at the time).