Trouble with conceptualizing the "temperature" of a deck of cards

The analogy is not well justified. In chemical systems, temperature can be introduced as $$T = \left(\frac{\partial U}{\partial S}\right)_{V,N}.$$ The constant specifiers are important. You can't introduce additional constraints like $U = \text{const}\cdot N$. And if energy was just constantly proportional to $N$, independent (explicitly) of $S$ and $V$, the above formula would give zero.

A situation analogous to yours in physics would be an ensemble of ordered particles, each of which can only be in a single level of an energy $\epsilon$. The only source of entropy is the combinatorics of the order. The only way the system can gain energy is by acquiring new particles. If it is closed, it has no way of exchanging heat with its surroundings, either way. There is no notion of temperature in such systems. They are, by definition, not hotter or colder than anything around them.

UPDATE based on comments.

OK, let us abstract from the "number of cards = number of particles" I have implicitly assumed and consider some imaginary system that can absorb energy in multiples of some quantum $ε$ and has $(U/ε)!$ microstates representing a macrostate of energy $U$.

This looks simple enough to analyze so in order to see how it behaves in thermal contact with another body, let us make an explicit calculation.

Consider a quantum-mechanical harmonic oscillator aligned such that $ħω = ε$, neglect the zero state energy. Let the compound system has, say, a total energy budget of 5 quanta. In a microcanonical situation, the following states have all the same probability:

1. oscillator at $5ħω$, deck of cards empty,
2. oscillator at $4ħω$, deck of cards has one card,
3. oscillator at $3ħω$, deck of cards has two, ordered AB or BA (2 distinct states),
4. oscillator at $2ħω$, deck of cards has three: ABC, ACB, BAC, BCA, CAB, CBA (6 distinct states),
5. oscillator at $1ħω$, deck of cards has 24 distinct states,
6. oscillator at $0ħω$, deck of cards has 120 distinct states.

This is a total of 154 microstates of equal energy that by assumption the system can transit between freely. Statistically, they all have equal probabilities in the long-term limit. Clearly, in most cases the oscillator is at zero, and nonzero energies have decreasing probabilities. Taking the partial totals: $$p_0 = 78\%,\ p_1 = 16\%,\ p_2 = 4\%,\ p_3 = 1\%, p_4 = p_5 < 1\%.$$

From the perspective of the oscillator, this looks somewhat like a thermal distribution at some temperature. (It would work better if I had many of them, but for this little demonstration this will suffice.) Let's see what happens if we pour some more energy into the system. Generalizing to $N$ total excitations:

Probability of $n$ excitations in the h.o., $N-n$ cards in the deck, $p_n = (N-n)! / \sum_{k=0}^{k=N} k!$

For $N\gg1$, the largest factorial by far dominates all the other terms in the sum combined, so let's simplify to $p_n \approx (N-n)! / N!$. Also, using Stirling's formula:

$$p_n \approx (N-n)! / N! \approx \left(\frac{N-n}{e}\right)^{N-n} \left(\frac{e}{N}\right)^N = \left(1 - \frac{n}{N}\right)^N \left(\frac{e}{N-n}\right)^n $$

In the first term we spot an approximant of the exponential ($n \ll N$):

$$p_n \approx e^{-n} \left(\frac{e}{N-n}\right)^n = \frac1{(N-n)^n} \stackrel{n \ll N}{\approx} \frac1{N^n} $$

Remember that Maxwell-Boltzmann distribution of the harmonic oscillator alone at temperature $T$ gives

$$p_n = (1 - e^{-ε/(kT)}) e^{-nε/(kT)}$$

and for $kT \ll ε$,

$$p_n \approx e^{-nε/(kT)} = (e^{-ε/(kT)})^n = \frac1{(e^{ε/(kT)})^n}.$$

Comparing the two, we see that for large $N$, the harmonic oscillator thermalizes at temperature $T$ for which $e^{ε/(kT)} = N$, which is

$$T = \frac{ε}{k \ln N}.$$

This would indeed confirm that in our model system the temperature is a decreasing function of the total energy $U = Nε$, or the total energy is a decreasing function of temperature, opposed to the usual. If we add some energy quanta to the system, they would rather find their way into the deck than into the oscillator, and even draw some more energy from the latter along, cooling it down.

Because this is an equilibrium, the same temperature would be ascribed to the deck of cards. Of course, the $N$ particles are split between the two. But for all practical purposes, $n \approx 0$ and $N-n \approx N$, so we can actually use this formula unchanged.

The explanation of this strange result is the superexponential explosion of the number of microstates of the deck of cards with energy. It is normal that this (and its logarithm, entropy) is an increasing function (in systems allowing saturation, entropy can start decreasing at some point; these then allow for negative temperatures), but in all common situations entropy stays concave. What is unusual about our "deck of cards" system is that it is a convex function in energy. This also means negative heat capacity, which brings in further paradoxes.

So, to answer your questions, there is no flaw in your logic; it is just a rather unusual model with very surprising properties. There may be something preventing it from being physical, I haven't tried to answer that.