How could I stop ssh offering a wrong key?

Solution 1:

This is expected behaviour according to the manpage of ssh_config:

 IdentityFile
         Specifies a file from which the user's DSA, ECDSA or DSA authentica‐
         tion identity is read.  The default is ~/.ssh/identity for protocol
         version 1, and ~/.ssh/id_dsa, ~/.ssh/id_ecdsa and ~/.ssh/id_rsa for
         protocol version 2.  Additionally, any identities represented by the
         authentication agent will be used for authentication.  

         [...]

         It is possible to have multiple identity files specified in configu‐
         ration files; all these identities will be tried in sequence.  Mul‐
         tiple IdentityFile directives will add to the list of identities
         tried (this behaviour differs from that of other configuration
         directives).

Basically, specifying IdentityFiles just adds keys to a current list the SSH agent already presented to the client.

Try overriding this behaviour with this at the bottom of your .ssh/config file:

Host *
IdentitiesOnly yes

Solution 2:

For me the solution was to add a key to a list of ssh keys, with a command:

ssh-add ~/.ssh/id_name_of_my_rsa_key

so it could be offered when connecting to the server. After adding a ssh, it was automatically recognized the correct one.

Edit:

But recently I think the better solution, and more permanent one, is to go to ~/.ssh/config and add IdentitiesOnly yes in your configuration file like this:

Host github.com
  HostName github.com
    User git
      IdentityFile ~/.ssh/id_rsa
      IdentitiesOnly yes