`ssh-agent` asks passphrase after it has been added

I just discovered that IdentitiesOnly seems to need the public key on the local machine to works properly.

Without the public key on the local machine, I was always prompted for the passphrase even if the private key was in the agent and of course the public key on the remote machine.

The public key path is the same as the private key path with .pub append.

As @ilkkachu said, the issue is IdentitiesOnly.

In you Host section, just add

IdentitiesOnly no