Design a filter that allows frequencies below 5 kHz to pass freely, but all frequencies above 5.2 kHz must be undetectable

You have assumed a 20dB/dec per filter order roll-off for your filter. This is not true for all filter types.

Let \$f_0 = 5 \mathrm{kHz}\$ and \$f_{\mathrm{stop}} = 5.2 \mathrm{kHz}\$. Then $$\frac{f_{\mathrm{stop}}}{f_0} = 1.04.$$

Have a look at this fourth order elliptic filter taken from the Wikipedia article.

Although it does not quite meet your requirements you can see it is feasible. A higher order elliptic filter can achieve what you are after.

You should keep in mind that elliptic filters can do disturbing things to the phase of the signal. Since you did not mention anything about your phase constraints, I have assumed that an elliptic filter is suitable.

This kind of sharp rolloff requires a digital filter. Don't even think about analog. You need to convolve the input with a sinc function. The width of the sinc function (the number of kernel points) defines the stop band attenuation.

I haven't done the math, but some very quick (could be off, your job to do properly) calculations says you probably need a few 100 points if sampling at 20 kHz. 200 points at 20 kHz means a MAC rate of 4 MHz. That's doable, in fact well below what modern DSPs can do rather easily. That means your problem is quite tractable. Something like a E series dsPIC can do this, and that's rather low end if you're only looking for DSP capability.

If you allow a substantial delay or process a recorded signal you can simply do FFT , remove unwanted components and invert the transform. You must truncate the fft with proper window function to keep the ringing acceptable.