sign_and_send_pubkey: signing failed: agent refused operation
So after hours of mindless googling and help, the problem was uncovered. I was generating my ssh keys with ssh-keygen and added an additional argument "-o" which generated the keys in a new format for openSSH. The problem was that my gnome-keyring did not support such keys as the keys had Ed255519 signature scheme. Gnome-keyring does not support that since 3.20. I reverted to RSA and no more problems!.
In my case the problem was that GNOME keyring was holding an invalid passphrase for the ssh key to be used. After spending indecent amount of time troubleshooting this issue I ran seahorse
and found the entry to hold empty string.
I can only guess that it was caused by mistyping the passphrase at first use some time earlier, and then probably cancelling the requester or so in order to fall back to command line.
Updating the entry with correct passphrase immediately solved the problem.
Deleting that entry (from "login" keyring) and reentering passphrase at that first prompt (and checking the appropriate checkbox) solves this too.
Now agent gets the correct passphrase from the unlocked at login keyring named "login" and neither asks for passphrase nor "refuses operation" anymore. Of course YMMV.