What's the best way to authenticate an user using a websocket?

If you want security, transmitting authentication data through TLS is a big start. But let's assume the websocket is already set up over TLS. A solution I would recommend would be based on a challenge-response mechanism :

1. The server would set up a random set of bytes (the challenge), set a timeout at the end of which the challenge wouldn't be authorized.
2. The client would send in "plain text" the username and the challenge and an hash (with a SHA2 or SHA3 algorithm) of challenge || username || f(password), f(password) being how the password is stored in the database (a SHA256 of password for instance).
3. The server verifies with the username's data and the challenge the response and if so, authenticates the user. It also invalidates the challenge to invalidate any future attempt using this challenge.

This method provides an anti-replay functionality : an attacker could not capture the data and replay it later to authenticate.

I however have one big problem with this method : f(password) has here the roughly same security as password : if the database leaks, any attacker could authenticate as anybody else. I say "roughly" because, if stored in clear text, since a lot of users are using the same password on multiple websites (which is bad), their accounts could be compromised elsewhere.

I can't think of a method protecting against a database leakage and an anti-replay mechanism, but an anti-database leakage mechanism (send user password in clear text over TLS, hash password with database salt and compare with database) over TLS should be sufficient enough against most risks you will face

If you want passwords, the best way is to use TLS, and then send the passwords in clear. This is simpler than a challenge-response mechanism.

If you however can't have TLS for websockets, but you have TLS for the code (delivered over HTTP), best thing to do is to do the login via HTTPS, generate session cookies (or some other form of keys) and use them to authenticate to the websocket. If you make challenge-response in the clear, an attacker can brute-force the password using that communication. With the session approach, an attacker can only capture the session of your (un-sensitive) game, but not the (possibly re-used) password. However when you have TLS on neither, you should let users authenticate through third parties, e.g. through OpenID. Then you don't need to handle sensitive data like passwords. OpenID is a good option for the other cases too.