"ssh-add -d" refused to remove identity

I had the same issue with Sierra. Try removing id_rsa from $HOME/.ssh/ and then restarting (I removed id_rsa.pub as well - therefore the two keys private and public). It solved my problem.

This error happened to me when the identity being referenced was different from the one that was added. The -d option removes a specific key. If that key was never loaded it refuses to remove it. You can check the keys loaded with ssh-add -l and check key signatures with ssh-keygen -lf <path-to-private-key>

Had the same Issue, fixed with

  killall ssh-agent

  ssh-add -l

The agent has no identities.