Apple - Enable ssh-key-only login to my Mac for 1 user

To disable password authentication you need to edit /etc/ssh/sshd_config. In Terminal:

sudo vim /etc/ssh/sshd_config

(or replace with your favourite editor)

Look for the line:

#ChallengeResponseAuthentication yes

And change it to:

ChallengeResponseAuthentication no

That is, remove the # at the beginning, and change yes to no.

All subsequent ssh logins will now REQUIRE you to use an ssh key and will not prompt for a password. Ensure your ssh key works before doing this if you have no local access! There's no need to restart the ssh daemon or anything because it is started on demand for each new incoming connection. Note that you may also see a PasswordAuthentication line, but that defaults to no already.

If your ssh keys aren't working then that's a separate issue really, but this answers the question as it is posed now :)

To update the answer for macOS Sierra (10.12.6 specifically), it appears the config file is now /etc/ssh/sshd_config

2 lines must be uncommented+modified to fully disable password authentication: (These are lines 57 through 62, inclusive, in the default sshd_config file)

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to no to disable s/key passwords
ChallengeResponseAuthentication no

As mentioned before, you may enable/disable ssh access from the System Preferences.app GUI, and no restart of the daemon is required between editing the configuration.