Does evaporation decrease entropy?

No, in fact you could even view the spontaneous evaporation as being driven by the fact that it increases entropy.

Basically what's happening is the liquid particles have random speeds (with distribution characterized by temperature), and they bump into each other. Every once in a while, two particles near the interface will collide in just such a way that one of the particles will escape the liquid and transition to the gas phase. The particle that remains in the liquid will loose energy in the process, thereby slightly lowering the temperature of the liquid. However, generally a gas has considerably more specific entropy (entropy per particle) than liquid at fixed temperature/pressure [1]. So, the overall entropy of the system increases.

[1] Compare for example the molar entropy for solid, liquid, and gas phases of water in these tables from Wikipedia.

It depends on how you define your system or your control volume. If only the container is considered then indeed the entropy has decreased due to cooling. On the other hand if you account for the container plus the escaped vapour the entropy has increased, as the randomness of the molecules in the vapour state is larger than compared to in liquid state at the same temperature. Instead of randomness you can picture entropy as the number (actually the logarithm) of possible states in the system that lead to the same macroscopic observables (as T, p, etc.).