Restrict password based SSH access per user but allow key authentication

Solution 1:

You can add "Match" sections to match on particular users or groups at the bottom of sshd_config, like:

Match user stew
PasswordAuthentication no

or

Match group dumbusers
PasswordAuthentication no

Solution 2:

Just lock the passwords of the users you don't want to log in with passwords:

usermod -L <user>

Then, place a valid public key in their .ssh/authorized_keys file and they will be only able to log in with the corresponding private key, but not with a password.

Note: This will break sudo unless the user has NOPASSWD: in their visudo entry