How to authenticate in a website with public/private keys?

Certificates can be used in that purpose. The general use of certificates is for TLS/SSL connections, but client certificates can be issued to handle two-way authentication. Certificates are basically public keys with an added layer of "trust networking".

If you want to do this properly, you should implement a PKI to handle certificate request, creation and revocation. But alternatively, you could just ask a user to upload any public key certificate they want to use.

Many recent browsers are also able to generate certificate request (private/public key) on the fly when connected to a website. Then the certificate are stored in the browser profile.

For people looking at this question now - this may be a better (updated) answer:

https://webauthn.guide/

From the site:

The Web Authentication API (also known as WebAuthn) is a specification written by the W3C and FIDO, with the participation of Google, Mozilla, Microsoft, Yubico, and others. The API allows servers to register and authenticate users using public key cryptography instead of a password.