Force the use of a gpg-key as an ssh-key for a given server

man ssh_config says about IdentityFile:

Additionally, any identities represented by the authentication agent will be used for authentication.

So it you set IdentityFile /dev/null, that one authentication will fail, then ssh will proceed to trying keys in your agent.

I can't do that because my identity is a gpg cardno.

You can use IdentityFile and IdentitiesOnly, even with gnupg-provided identities.

• If you have the card present, export the public key from your agent:

  $ ssh-add -L | grep "cardno:.*789$" | tee ~/.ssh/smartcard.pub
  ssh-rsa AAAA[..]== cardno:023456000789
  

• If you do not, but remember which key it is associated with, export from gnupg:

  $ gpg2 --export-ssh-key [email protected] | tee ~/.ssh/smartcard.pub
  ssh-rsa AAAA[..]== openpgp:0xDEADBEEF
  

Then tell ssh to use that export to identify the correct key:

Host *.host.example
    IdentityFile ~/.ssh/smartcard.pub
    IdentitiesOnly yes
    PasswordAuthentication no
    PubkeyAuthentication yes

Which gives you exactly one login attempt as expected when the correct smart card is detected by gnupg:

$ ssh -v smart.host.example
[..]
debug1: Next authentication method: publickey
debug1: Offering public key: /home/home/.ssh/smartcard.pub RSA SHA256:a1337[..] explicit

Unfortunately, you get rather unhelpful output whenever you forget to insert the card, as the gnupg ssh agent will not ask to insert the correct card like the gpg agent does. This is annoying, but will not impact your actual use.