SSH closes connection when key is accepted
The actual problem was that my user was somehow missing from `/etc/shadow.
I will let this question to stay open for sometime so that people can put more troubleshooting advice here.
Also, make sure the permissions on ~/.ssh are 700 and ~/.ssh/authorized_keys is 600. The server won't let you use the key if the permissions are incorrect.