Why is argon a noble gas but not, say, beryllium or palladium?

Aufbau arises mainly from the fact that the screened potential is not the same as the $1/r$ potential of hydrogen. In the screened potential, some levels are pushed up in energy and some are pushed down. Basically the high-l orbitals have less probability concentrated near the nucleus, where the screened potential is the most negative, so they're pushed up in energy relative to the low-l orbitals.

In addition to this, you get effects that are the way they are not because of any semiclassical argument because of the way the Schrodinger equation works. You're basically diagonalizing a big matrix, and there is no guarantee that the results will come out to be easy to explain. Sometimes such results can be at least partially described by heuristics, such as a tendency for energy levels to "repel" each other, although it's not actually a physical repulsion. For this "level repulsion" to happen, the levels have to have the same angular momentum. (It's actually pretty surprising that this kind of level repulsion leaves the $n$ shells degenerate in hydrogen. You would think it would split them up.)

Noble gases exist when there is a large gap between the single-particle energy levels. These gaps do not occur for the screened potential at the same particle numbers where they would occur for hydrogen.

Here are some energy levels of states in noble gases, calculated using the Hartree-Fock method, from Johnson, Lectures on Atomic Physics, p. 86:

The Hartree-Fock method calculates the energies self-consistently for a particular atom, so the energies are different for the different atoms. It would be convenient to have the calculation for something like potassium, so we could see the gap between the last filled state in argon and the final electron for potassium, but he doesn't actually give those. However, you can clearly see the relevant shell gap in the calculation for krypton. There is a gap of 4.5 Ha between 3p and 3d. This is considerably larger than the gap between 3s and 3p (2.5 Ha) and between 3d and 4s (2.7 Ha).

So - what is special about p sub shells such that when they fill there is a larger gap up to the next level than when s,d, or f sub shells fill?

There is nothing special about p shells. These calculations are based on diagonalizing a big matrix. There is no particular reason to expect the spacing between subshells to follow any simple rules.

Beryllium isn’t a noble gas.

If you look at the levels for argon, you can see that the gap between 2s and 2p is less than 3 Ha, while the gap between 2p and 3s is about 8 Ha. The reason for the sizes of the gaps is ultimately just that that's what you get when you diagonalize the matrix. However, I don't find the result particularly surprising, since the unscreened potential has 2s degenerate with 2p, and a big gap between 2p and 3s.

As a proxy for an atom's reluctance to form chemical bonds, we can use the energy required to lift one of its outer electrons to the next higher orbital, pretending as usual that the "orbital" concept is still at least approximately valid in a multi-electron atom. Then the question is why this energy gap tends to be significantly larger after filling a p-shell ($\ell=1$) than it is after filling s- or d- or f-shells ($\ell=0,2,3$).

As usual, let $n$ and $\ell$ denote the radial and angular quantum numbers, respectively. Let's accept that orbitals are filled in order of increasing $n+\ell$, and then in order of increasing $n$ whenever the first rule is neutral. (Reality is a little more complicated, but these rules mostly work pretty well.) The following list shows the filling order defined by these two rules, along with the number of nodes in the orbital wavefunction's radial factor: \begin{align} n+\ell &\hskip1cm (n,\ell) &\hskip1cm \text{radial nodes} & \\ \hline 1 &\hskip1cm (1,0) &\hskip1cm 0 &\hskip1cm \text{helium}\\ 2 &\hskip1cm (2,0) &\hskip1cm 1 \\ 3 &\hskip1cm (2,1) &\hskip1cm 0 &\hskip1cm \text{neon}\\ 3 &\hskip1cm (3,0) &\hskip1cm 2 \\ 4 &\hskip1cm (3,1) &\hskip1cm 1 &\hskip1cm \text{argon} \\ 4 &\hskip1cm (4,0) &\hskip1cm 3 \\ 5 &\hskip1cm (3,2) &\hskip1cm 0 \\ 5 &\hskip1cm (4,1) &\hskip1cm 2 &\hskip1cm \text{krypton} \\ 5 &\hskip1cm (5,0) &\hskip1cm 4 \\ 6 &\hskip1cm (4,2) &\hskip1cm 1 \\ 6 &\hskip1cm (5,1) &\hskip1cm 3 &\hskip1cm \text{xenon} \\ 6 &\hskip1cm (6,0) &\hskip1cm 5 \\ 7 &\hskip1cm (4,3) &\hskip1cm 0 \\ 7 &\hskip1cm (5,2) &\hskip1cm 2 \\ 7 &\hskip1cm (6,1) &\hskip1cm 4 &\hskip1cm \text{radon} \\ 7 &\hskip1cm (7,0) &\hskip1cm 6 \\ 8 &\hskip1cm (5,3) &\hskip1cm 1 \\ \end{align} In every noble-gas case, and only in these cases, the next available level is an s-orbital ($\ell=0$), which has more radial nodes than the levels immediately before or after it in the sequence. (This pattern works for helium, too, even though helium doesn't have a filled p-shell.) If we could understand intuitively why energy gap between the ground state and the first excited state is relatively large when the first excited state is an s-orbital, then we would have at least a partial answer to the question.

The order in which the shells are filled (the sequence shown above) indicates that radial nodes are more costly than "azimuthal" nodes, because for a given $n+\ell$, the cases with smaller $\ell$ (fewer "azimuthal" nodes) have higher energy. Accepting this trend as an axiom, we can focus our intuition on the radial part.

Intuitively, if we think of the radial nodes as "no-fly zones" for that electron, then having a larger number of radial nodes may correspond to having less freedom to re-arrange the multi-electron system to minimize the energy in the presence of electron-electron interactions. Here's the key idea: by analogy with a construction-induced traffic jam, the impact of each additional node (each additional no-fly zone) could be an increasing function of the number of nodes already present. This picture suggests that lifting an electron from a filled p-shell up to the next available s-shell should be more costly than, say, lifting an electron from a filled d-shell up to the next available p-shell, because the former lift requires increasing an already-larger number of radial nodes. Using the magnitude of the energy-gap as a proxy for an atom's reluctance to form bonds, this suggests that the noble gases should be less reactive (relatively), at least among atoms with the same value of $n+\ell$.

The key idea is that the energy-cost of each additional radial node is an increasing function of the number of radial nodes already present. This seems to be consistent with the information I've seen, but I have no real justification for anticipating it except for the dubious traffic-jam analogy. Even if the intuition is correct, it isn't quantitative enough to predict just how noble the noble gases are. It is suggestive at best, but at least it doesn't rely entirely on a computer. That's why I thought it was worth posting.

In my opinion @user4552's answer captures the key points from a theoretical perspective:

1. The reactivity of an atom is determined by it's energy level structure.
2. A simple semi-classical model (like Bohr's model) is insufficient to capture the full energy level structure, due to screening effects and interactions -- i.e. spring orbit coupling. The Hartree Fock method yields satisfactory results.

Nevertheless, I feel like we should point out that the name "nobel gas" was used before we understood the physics. Thus, the distinction becomes somewhat arbitrary, if the electron number becomes "large". In order to emphasise this argument, I took the ionisation energies (from NIST database) and plotted them as a function of the number of electrons (=#protons): By interpreting this graph see that

• the electrons of nobel gases are "strongly" bound to their nucleus (="large" ionisation energy). Hence, nobel gases "do not like to share" one of their electrons.
• nobel gases "do not like to accommodate" an additional electron. This can be seen by considering the atom which has one additional proton. These (alkali) atoms "like to get rid" of their additional electrons -- alkali atoms are "easy" to ionise. Thus, by adding an additional proton to a nobel gas atom does not increase the ionisation energy.

Combining these two observations, the ionisation energy is a simple measure to understand nobel gases: Neither do accommodate an additional electron, nor do they share an electron with other atoms. Thus, they are "reluctant" to react with other atoms.

In the graph I mark the nobel gases in red. We see that the ionisation energy increases steadily towards the nobel gases, it reaches it's local maximum at the nobel gas atom, and then takes an abrupt, negative step. However, this structure is also visible for the atoms with filled $d$-shell (marked in green): Here we also observe a steady increase in the ionisation energy, a local maximum followed by an abrupt, negative step. Thus, the distinction between nobel gases and atoms with filled $d$ shells is not significant. Furthermore, with increasing electron number both (a) the ionisation energy and (b) the step size in energy become approx. equal. Hence, calling radon a nobel gas and mercury a transition metal is a matter of convenience (consistency with the historical system/nomenclature) and not a statement of "stability" (in my opinion).

Using intuitive, semi-classical arguments to explain quantum effects is a sure path towards failure. Nevertheless, since Slater has already done this, there is little harm to restate the idea. Slater modelled the screening of the electrons by $$ E = - h c R_H \left(\frac{Z-S}{n^\star}\right)^2 $$ where $R_H$ is Rydberg's constant, $S$ is a screening factor and $n^\star$ is an effective quantum number. Now, the screening factor depends on the number of electrons within a single shell. (simplified)

• Helium: Before ionisation there are two $s$-shell electrons. Thus the screening factor is $1/2$. After the ionisation the screening factor is zero. The difference of these screening factors determines how strong the electrons are bound.
• Lithium: Before ionisation there are three $s$-shell electrons. This yields a screening factor $3/2$. After the ionisation the screening factor is $1/2$.

Using this kind of argument, we see that the $d$-shell electrons are strongly screened. Because a filled $d$-shell contains ten electrons, the screening difference between ten and nine electrons is "rather small". In contrast, the filled $p$-shell contains only six electrons. Thus, the difference in screening effect is larger as in the $d$-shell.

Using Slater's argument, one could conclude that the filled $s$-shells should form the nobel gases. So it's only an argument against the $d$ and $f$ shells.