How can I make a user able to log in with ssh keys but not with a password?

Use of passwd -d is plain wrong , at least on Fedora, on any linux distro based on shadow-utils. If you remove the password with passwd -d, it means anyone can login to that user (on console or graphical) providing no password.

In order to block logins with password authentication, run passwd -l username, which locks the account making it available to the root user only. The locking is performed by rendering the encrypted password into an invalid string (by prefixing the encrypted string with an !).

Any login attempt, local or remote, will result in an "incorrect password", while public key login will still be working. The account can then be unlocked with passwd -u username.

If you want to completely lock an account without deleting it, edit /etc/passwd and set /sbin/nologin or /bin/false in the last field. The former will result in "This account is currently not available." for any login attempt.

Please refer to passwd(1) man page.

Are not you asking specific to SSH daemon, not to accept the password based authentication but the key/passphrase authentication?

Look for changes to sshd_config.

Set

PasswordAuthentication No
PreferredAuthentications publickey,hostbased,keyboard-interactive
Protocol 2,1

Look for more config parameters at man ssh_config