How do we know that the nucleus isn't a quark-gluon plasma?

Because the interaction energies inside the nucleus are very low, and when we perform experiments on nucleons at those energies they behave primarily as discrete particles.

For a particular instance, take quasi-elastic nucleon ejection reactions like $A(e,e'p)$ (that is an electron scattering off a nucleus and neatly knocking out one proton without otherwise exciting or shattering the remnant). The figure of merit for such reactions is $Q^2 = -q \cdot q$, and in the range from about $1$--$8\text{+ GeV}^2$ (the upper limit is not yet experimentally known) the cross-section for this (per target proton) is a constant times the cross-section for the free reaction $p(e,e'p)$. The constant is called the "nuclear transparency" and expresses the probability that the scattered proton re-interacts before leaving the nucleus.

See this figure from my dissertation work

The nuclei (shown by different marker shapes and colors) are deuterium, carbon, iron and gold, and the different marker fill-states and sizes represent three experiments. Error bars are shown for all data and represent statistics plus point-to-point systematics (but not not systematics that would shift all the data for a single nucleus). The solid lines are a modified Glauber calculation.

A big part of what they do at Jefferson laboratory is to search for the change between nucleon--meson degrees of freedom (as reasonably well described by quantum hadrondynamics) and quark--gluon degrees of freedom (as described by quantum chromodynamics); but even the on-set of QCD dominated behavior does not correspond to quark--gluon plasma which occurs at much higher energies.

What is the evidence that large nuclei have nucleons as substructures?

The periodic table of elements? This shows an organized internal substructure into nucleons, i.e. triplets of quarks in SU(2) representations:

The atomic masses in the table are within a percent multiples of the nucleon mass (the binding energy takes up the slack) . A quark gluon plasma would not show this approximate quantization in multiples of triplets.

A quark gluon plasma would not be constrained this way.

There are experiments looking for the signatures of a quark gluon plasma. Here is a review on such signatures.

dmckee covers the topic from an experimental perspective, but you can also approach the issue from the theoretical perspective of quantum chromodynamics (QCD), which explains why the experimental results turn out the way that they do.

One of the core principles of QCD is "confinement" which means that the strong force between quarks that is mediated by gluons is so pervasive that none of the five kinds of quarks that are not top quarks (top quarks decay so quickly they don't have time to form composite structures which is called hadronizing before they decay) are ever observed in nature outside of a "hadron" (a composite particle made of quarks bound by gluons) - generally either three quarks (one of each of the three QCD color charges) in a "baryon" or a quark and an antiquark bound in a "meson". Baryons are "fermions" of spin $-1/2 + N$ (an integer), which means that they behave like ordinary matter (i.e. to oversimplify, you can't have two of them in the same place at the same time), while "mesons" have integer spin N so, to oversimplify, more than one can be in the same space at the same time.

In principle, QCD allows four or more quark composite particles (tetraquarks, pentaquarks, etc.) and a handful have been observed, but in practice, the four or more quark composite particles are extremely unstable and hard to create in the first place, so even in the rarified environment of a particle collider you see almost entirely mesons and baryons.

A quark-gluon plasma can "fudge" the otherwise rather automatic and complete sorting of quarks into confined hadrons only at extremely high energies because the strong force powerfully constrains them to stay within a particular hadron. So, in the "infrared" environment that we encounter in daily life or even in quite high energy applications, the temperature isn't enough to overcome the strong force's tendency to bind quarks into hadrons.

How hot does it have to be?

The cross-over temperature is about $2 \cdot 10^{12} \mathrm K$ which corresponds to an energy density of a little less than $1 \mathrm{GeV/fm^3}$ (i.e. the temperature has to contribute kinetic energy density comparable in amount to the energy density of the gluons in a proton or neutron to overcome their clutches). This is more than one hundred times as hot as the hottest it gets anywhere inside the Sun (which is about $1.5 \cdot 10^{10} \mathrm K$).

How hard is it to get something that energetic?

The first time humans were able to artificially create those energy densities was in 2015 at the Large Hadron Collider (although non-definitive hints that we might have done it were seen at other colliders as early as 2005). Nothing in the solar system had ever been that hot previously at any time in the four or five billion years since it came into existence.

Now, it also turns out that of the hundreds of possible baryons and mesons, no meson has a mean lifetime of more than about a ten millionth of a second, and only two kinds of baryons have mean lifetimes of more than about ten billionths of a second. The proton (which is made up of two up quarks and one down quark bound by gluons) does not decay, and the neutron (which is made up of one up quark and two down quarks) has a mean lifetime of about 15 minutes if free (and can potentially be stable in the right kind of atomic nucleus mostly due to conservation of mass-energy considerations in a bound nucleus).

So, the theoretical reason that atomic nuclei are not quark-gluon plasmas is because (1)(a) at cool enough temperatures and (1)(b) given the tiny fraction of a second necessary for an unstable hadrons to decay, (2) their constituent quarks are forced into hadrons and (3) the hadrons decay until they consist only of stable protons and neutrons (collectively nucleons).

A residual effect of the strong force (which is mediated mostly by a kind of meson called a pion which is exchanged between nucleons) binds the nucleons together into an atomic nucleus, but far less tightly to each other than the quarks within the nucleons are bound by the strong force.

Now that binding force itself is nothing to sniff at - that is the source of all energy created by nuclear fusion in an H bomb or a star like the Sun, and the nuclear fission in an A bomb or nuclear reactor. But, the strong force binding quarks inside protons and neutrons is much stronger which is why it takes such extreme conditions to overcome.

• Caveat: There are multiple, progressively more complex ways to describe what is going on in a hadron. In addition to the "valance quarks" bound exchanging gluons that I have described above, there is also a "sea" of quark-antiquark pairs that cancel out for many purposes but can't be ignored if you want to smash protons together at high speeds or you want to calculate the mass of a hadron from first principles. Thus, for example, you could smash protons only to find that strange or charm or bottom quarks fall out if the collision has enough energy, even though none of the valence quarks of a proton are of that type. But, that level of complexity isn't necessary to understand theoretically based on QCD why quarks in an atomic nuclei are subdivided into protons and neutrons.