Why are protons and neutrons the "right" degrees of freedom of nuclei?

This is basically a matter of energy scales. By analogy, you could ask why we don't take into account nuclear structure when we talk about chemistry. The answer is that the eV energy scale of chemistry is mismatched with the MeV energy scale of nuclear structure.

Nuclear matter has two phases. One is the phase we normally see, and the other is a quark-gluon plasma. The phase transition happens at a temperature on the order of 100 GeV per nucleon (at standard nuclear densities). Below the temperature of the phase transition, the quarks are strongly correlated, and those correlated groups behave in a way that's very similar to free neutrons and protons. To the extent that they don't quite have those properties, often we can subsume the discrepancies within adjustments to the parameters of the model. It's helpful in terms of practical computation that the fictitious neutrons and protons are nonrelativistic, which makes the theory much more tractable than QCD. If there are small relativistic effects, because the nucleons are moving at a few percent of $c$, these can also be subsumed within adjustments to the parameters.

By the way, it is actually possible to consider larger clusters to be the relevant degrees of freedom for nuclear structure. There is a model called the interacting boson approximation (IBA, also known as the interacting boson model, IBM), in which pairs of nucleons coupled to spin 0 or 1 are considered the degrees of freedom. It does pretty well in phenomonologically fitting the properties of many nuclei that are intractable in other models.

In a similar vein, there are alpha cluster models and ideas like explaining alpha decay in terms of preformation of an alpha particle, which then tunnels out through the Coulomb barrier. Pictures like these go back to the 1940's, and have considerable utility and explanatory power, although they can't really be microscopically correct, because they violate the Pauli exclusion principle.

We can measure the form-factors of bound nucleons. For instance by doing quasi-elastic scattering of a proton out of the nucleus $A(e,e'p)$ at low energy loss (my dissertation work involved this reaction for deuterium, helium, carbon and iron).

The result are quite similar to (but measurably not identical to) the equivalent results on free protons. That similarity make the choice of nucleons as the degrees of freedom a good starting point.