Does higher impedance speakers generate clearer voice?
With the higher impedance (8 ohm) speaker, you will deliver 3dB less power to the speaker than you would with a 4 ohm speaker. However, the efficiency of the speaker will likely have a much larger effect on the volume. In general, speakers with lower efficiency will have less distortion. Note that typical efficiency for speakers ranges from about 4% for loud, cheap ones to 0.2% for reasonably good audio equipment. (Of course, it's easy to make a speaker that's both distorted and inefficient, so it's not a hard rule.)
Amplifiers will show higher distortion for lower impedance speakers because the higher currents required to drive them stress the drivers more, introducing higher levels of nonlinearity.
If you're looking to reproduce legible voice, you can easily handle the levels of distortion that you're looking at in this example; you probably wouldn't even notice it if you weren't listening for it. If you were looking to listen to music, my advice would be different.
The PAM8403 data sheet (link to datasheet) doesn't seem to contain anything to back up what you've read.
Look at the characteristic labelled "Total Harmonic Distortion Plus Noise". For the specified noise level, it can deliver twice as much power into a 4 ohm load as into an 8 ohm load. That implies you would be better off using a 4 ohm speaker and turning down the volume a bit, to reduce distortion further.