How does SSH encryption work?

First thing after establishing the TCP connection, both systems agree on a session key, using such protocols as DH key exchange, ECDH or GSSAPI. This key is symmetric and temporary – both sides use the same key to encrypt and decrypt data using such algorithms as AES or RC4.

The client keypair is never used for encrypting data, only for authentication – "publickey" is one of several available methods, where the client presents its own public key along with proof of private-key ownership. Similarly, the server keypair is only used for authenticating the server during DH or ECDH key exchanges; no data is encrypted using it.

The SSH2 protocol is documented in several RFCs, including:

• RFC 4253 – Secure Shell (SSH) Transport Layer Protocol
  • section 7 – "Key Exchange"
  • section 8 – "Diffie-Hellman Key Exchange"
• RFC 4419 – Diffie-Hellman Group Exchange
• RFC 4432 – RSA Key Exchange
• RFC 4462 – GSSAPI Authentication & Key Exchange

First thing I think you need to understand is that while many encryption protocols like SSH and SSL use PKI for authentication purposes, almost none of these systems will use PKI for actually transmitting the payload.

PKI is far too CPU intensive to be used for transmitting the actual payload data. What happens is that the PKI is used to negotiate a randomly generated key, to be used with a symmetric encryption protocol. The protocol to be used is also negotiated, and should be the strongest protocol the two systems can agree on. So once the initial handshake and negotiation is done, pretty much everything is just standard symmetric cryptography.

Here are some practical examples, Assume Key A was kept a secret and is therefor the private key and Key B was posted in a publicly accessible place and therefor is the public key.

So if you want to send a message to everyone and you want them to verify that it came from you and was unaltered while it was being delivered, you would send your message and include a hash of the message encrypted with Key A. Then anyone who has Key B can decrypt the hash, compare it to the message they received, and verify that the message came from you (due to the fact that only a person with Key A could have generated the encrypted payload that successfully decrypted hash, and because you are the only person with Key A it could only come from you). This is called Signing.

Now lets say someone wants to send you a secret message but does not want to reveal who they are. They can encrypt their message with a symmetric key (as Zoredache mentioned symmetric is much cheaper to do) then take that key and encrypt it with Key B and send it to you. Because only Key A can Decrypt something that was encrypted with Key B no other person can see what is in the message that was sent to you. This is how normal encryption works and how SSH exchanges data.