Energy conservation in reflection of light from a perfect mirror

However if I do assume each photon gives up a bit of it's energy, thus changing it's wavelength, then the incoming momentum will not be equal to the outgoing momentum. But this would lead to a contradiction as we assumed the mirror was perfectly reflecting.

There is no contradiction. You should interpret "perfectly reflecting mirror" to mean that every photon incident upon the mirror is reflected. It is indeed the case that this requires the reflected photons to have longer wavelength than the incident photons. You aren't given the reflected wavelength, but imposing energy-momentum conservation (two relations) allows you to solve for both the reflected wavelength and the value of $N$ (two unknowns).

If you ask "what if we imagined a perfectly reflecting mirror" in the sense of a mirror that would always reflect each incident photon without changing its wavelength, then you are effectively asking "what if energy or momentum was not conserved?" Well, then you'd have to propose some new laws of physics that would make this possible, and would hopefully answer your question.

In the problem statement "perfect mirror" must mean that no light is absorbed and that the mirror is perfectly flat. However, there is a recoil effect that is measurable if the mirror is light enough. This is required by momentum conservation, as you state. Every photon will therefore be reflected with a slightly smaller energy, hence a slightly smaller frequency and larger wavelength. In this sense the mirror is not perfect, by design. Your suspicion is well founded.

It would have been more clear if it had been explicitly stated what was meant by "perfect".

You are right to question the assumption that the impulse given to the mirror is twice the incoming photon's momentum, but this is not anything to do with the mirror being a perfect reflector. This is an approximation. You are correct that if the reflected photon's momentum is equal to the incoming photon's momentum, then the mass of the mirror must be infinite (or else, the reflected photon's momentum must be lower). This is another way of saying that the mirror cannot move. The best way to explain is to just do a simple kinematic calculation.

Let's ignore the spring and do a simple elastic collision of mirror and photon. This is a non-relativistic calculation (nothing like Compton scattering), so let's use non-relativistic energy and momentum conservation.

Let's take the incoming photon's wavelength to be $\lambda$, the reflected photon's wavelength to be $-\lambda^\prime$ (negative since it is reflected in direction), and the mass of the mirror as $M$. After the photon reflection, suppose the mirror is imparted a velocity $v$. By momentum conservation,

$$ \frac{h}{\lambda} + \frac{h}{\lambda^\prime} = M v \:, $$

and by energy conservation,

$$ \frac{hc}{\lambda} - \frac{hc}{\lambda^\prime} = \frac{1}{2}Mv^2\:. $$

You can convince yourself that after eliminating $h/\lambda^\prime$ and some rearranging you will have an equation that is quadratic in $v$, whose formal solutions will be

$$ v = -c \pm c\sqrt{1+\delta} \:,\:\:\:\:\:\:\textrm{where}\:\:\:\delta = \frac{4 h}{M c \lambda} \: . $$

I'm skipping a lot of trivial algebra (and the quadratic formula); you should be able to get the result above without too much trouble. We can immediately throw away the unphysical tachyonic solution, and, since $\delta \ll 1$, we can expand in $\delta$ to get

$$ \frac{v}{c} = \frac{1}{2}\delta + O(\delta^2) \:. $$

Thus, we get

$$ Mv \approx \frac{2h}{\lambda} = 2 p \:, $$

where we have ignored terms that are higher order in $h/\lambda$ (meaning higher order $\delta$ terms). Thus, the momentum of the mirror is approximately just twice the incoming photon's momentum. In other words, you can just approximate the kinematics of the system as if the reflected photon has the same momentum $p$ as the incoming photon, and that the mirror is therefore given a momentum $2p$ because the photon gets reflected (the photon's impulse must be $-2p$ in order to reverse direction, so the mirror's impulse must be $+2p$ in order to conserve momentum).

In reality, the photon will see some wavelength shift, but it will be small. The leading order term in the mirror's impulse will come from the photon's change of momentum due to reflection. Intuitively, this is because the rest mass energy of the mirror is much larger than the photon's energy. For the sake of intuition, you can pretend the photons are equivalent, here, to small particles of mass $m$, where $m$ is given by $m c^2 = hc/\lambda \ll M c^2$. Think of bouncing a marble on the ground, where the mass of the Earth is much larger than the mass of the marble: each individual photon's momentum will not change in magnitude by much since the mirror's mass is much higher, only its direction will change. This intuition is supported by the analysis above: we would expect our conclusions to break down when $\delta \sim 1$, or in other words, when $h/\lambda \sim Mc$ (ignoring trivial numerical factors).

As an aside, we can also approximate what the wavelength shift will be. The value of $v$ up to first order corrections will be

$$ \frac{v}{c} = \frac{1}{2}\delta - \frac{1}{8}\delta^2 + O(\delta^3) \: . $$

Thus, $$ Mv \approx 2 \frac{h}{\lambda} - \frac{2}{Mc} \left(\frac{h}{\lambda}\right)^2 \: . $$

Placing this expression back into to conservation of momentum equation at the top, we will have

$$ \frac{h}{\lambda^\prime} - \frac{h}{\lambda} \approx - \frac{2}{Mc} \left(\frac{h}{\lambda}\right)^2 \: . $$

So,

$$ \frac{\Delta p}{p} \approx - \frac{2p}{Mc} \:, $$

where higher order corrections in $p$ will be suppressed by factors of $1/Mc$. In terms of $\lambda$, this shift will be, up to first order corrections,

$$ \frac{\Delta\lambda}{\lambda} \approx \frac{2h}{Mc\lambda} \:. $$

So, if we take visible light (say, $\lambda = 5 \times 10^{-7} \,\textrm{m}$), and $M = 0.1 \,\textrm{kg}$, this proportional shift will be about

$$ \frac{\Delta\lambda}{\lambda} \approx 8 \times 10^{-35} \:, $$

which is absolutely the textbook definition of negligible. Where the mirror picks up its detectable motion is in the sheer number of photons hitting it.