Why do spectrum analyzers use envelope detectors?

After mixing and passing through the IF filter what remains is largely one small part of the spectrum that your SA is scanning through. If there's a signal present in this part of the spectrum, it'll be a sinewave and if you envelope detect it you are, in effect, measuring the peak amplitude of that sine wave. That peak signal is \$\sqrt2\$ higher than the RMS value so it's perfectly usable.

EDITED SECTION

There will be errors in assuming that the signal detected is a single sinewave. For instance if you use an averaging filter after the envelope detector and looked at what it told you when a single sinewave is in that part of the spectrum it would produce a reading that is 3 dB higher than the RMS.

If on the other hand you had three sinewaves at 999 kHz, 1000 kHz and 1001 kHz it would produce a level that is only 1 or 2 dB above the true RMS value and this does lead to a small error.

This block diagram is for an old-style CRT-based spectrum analyzer. The horizontal sweep represents the frequency being measured at a given time (like a sawtooth wrt time), and the vertical amplitude is the frequency content at that frequency (more accurately, over a narrow range of frequencies).

If the input signal is a modulated carrier the display will show the sideband(s).

This block diagram shows and old-style analogue spectrum analyser.

It uses a peak detector at that point for convenience. As Andy says, the peak signal is \$\sqrt{2}\$ higher than the RMS value, but my emphasis for a sinewave.

Unfortunately, a lot of the things a spectrum analyser has to measure are not sinewaves. One thing is noise. Another is complex radio communications modulation like OFDM. Modern spectrum analysers dispense with the final narrow IF filter, logamp and envelope detector, and digitise in a fairly wideband IF. They then channel filter digitally, and do true power detection in each channel. This is far more useful than peak detection, which has different ratios to the RMS for different waveforms.