Can One Public Key be Used to Encrypt and Decrypt Data during the SSL Handshake?
In a PPK algorithm like RSA, you have two different communication channels. Information encrypted using the public key is only readable to the possessor of the private key, and information encrypted using the private key is only readable to the possessor of the public key.
In fact, the selection of which half of the pair is "public" is completely arbitrary.
Now, in practice this doesn't much matter; the whole world has access to the public key, so encrypting something with the private part wouldn't do anything to secure it. But you can use this for authentication: since only one holder has the private key, if a message is validly encrypted using it, then the private-key holder must have been the author.
That's why your book doesn't say that the private key is used for encryption: because it's used for integrity, not for confidentiality, as any message sealed using it would be readable to anyone possessing the non-secret public half. While the integrity validation mechanism is technically encryption (it is encipherment using modular exponentiation), it would be confusing to mention this in a foundations-of-cryptography context as it's not what people think when they hear "encryption" - they think "privacy".
In public key cryptography:
- The private key is used for signing and deciphering/decrypting.
- The public key is used for verifying signatures and enciphering/encrypting.
See the glossary of the TLS specification:
public key cryptography: A class of cryptographic techniques employing two-key ciphers. Messages encrypted with the public key can only be decrypted with the associated private key. Conversely, messages signed with the private key can be verified with the public key.
You cannot encrypt with a private key or decrypt with a public key, not for mathematical reasons, but because it doesn't make sense w.r.t. the definition of encrypt:
To convert ordinary language or other data into code; to hide the meaning of a message by converting it into a form that cannot be interpreted without knowing the secret method for interpretation, called the key; to encode.
In a situation where you "encrypt with the private key", you effectively "scramble" the data indeed, but what's required to turn back the message into its original form is not a secret. Hence, it doesn't make sense to talk about encrypting in this context. Whether the mathematical operations behind it may work one way or the other doesn't matter at this stage.
Similarly, you don't really sign something with a public key. You only need one of the keys to sign, and that's the private key. You verify the signature with the matching public key.
It's quite common (even in the TLS specification) to say "signing with a certificate", when what's really implied is computing the signature with the private key matching the certificate. In many cases, not specifically TLS, the certificate itself is communicated along with the signature (whether-or-not one chooses to trust that certificate is another matter).
Expressions such as "using your certificate to authenticate" or "using your certificate to sign" are generally acceptable, so long as you understand that "certificate" is used there to shorten "certificate and its private key", and that it's in fact the private key that's necessary for those operations.
I don't have the book you're quoting, but this quote sounds misleading or incorrect (perhaps taken out of context here):
...the public key in the server's certificate will only be used to verify its(server) identity.
The public key in the server certificate isn't used to verify the server's identity itself. What it does is ensuring that only someone/something with the corresponding private key will be able to decipher what you've encrypted with this public key: in this case (authenticated key exchange), the pre-master-secret which the server will prove it knows to the client by producing the correct Finished message, based on the pre-master-master it has managed to decipher.
The identity binding is done by the certificate itself, with is the signed combination of the public key, some identifiers (e.g. Subject DN and Subject Alternative Names) and possibly various other attributes (e.g. key usage, ...). This side of the identity verification (i.e. checking who this certificate belongs to) is established by verifying the integrity of the certificate and that you trust what it says (usually PKI), and by verifying that the identity it belongs to is indeed the one you wanted to connect to (host name verification). This is done by verifying the certificate signature itself using a Certification Authority (CA) or by an outside mechanism, for example if you have explicitly granted an exception for a given certificate (possibly self-signed) using knowledge you have outside the scope of the PKI to which the certificate belongs. This step is rather independent of the TLS specification, although you'll need all these pieces together to make the communication secure.
There's a similar problem with this quote (again, possibly taken out of context):
...the key information is signed using the public key contained in the server's certificate.
Although saying "signed with a certificate" is a common expression (as explained above), I'd say "signing using the public key" is definitely confusing, since "public key" is normally used in contrast to "private key", and it's really the private key that's used for signing. While even the TLS specification (Section F.1.1.2) talks about "signing with a certificate" in a few places, "signing with a public key" is an unusual and misleading expression.
I'm not sure whether "(decrypt)" and "(encrypt)" are in the book or your additions:
the public key in the server's certificate can be used to verify(decrypt) the server's identity and sign(encrypt) the key information(,then the client will use the key information to encrypt pre_master_secret)
You actually verify you're talking to the actual server identified by that certificate because it's the only one capable of deciphering what you've encrypted with its public key (in the client key exchange message).
As it's put in the TLS specification Section F.1.1.2:
After verifying the server's certificate, the client encrypts a
pre_master_secret with the server's public key. By successfully
decoding the pre_master_secret and producing a correct finished
message, the server demonstrates that it knows the private key
corresponding to the server certificate.
What you're asking at the end doesn't completely make sense:
I know the public key can be used to verify server's identity(Certificate message), but I can't understand the public key why can be used to sign the key information, because the client doesn't have the corresponding private key, how does the client verify the key information?
The public key isn't used to verify the server's identity. You verify that you're talking to the server who has the private key matching the certificate it presented earlier by the fact it was able to decipher the pre-master-key and produce the correct finished message.
EDIT 2:
Following your edit, it seems that you're still using "sign(encrypt)" and "verify(decrypt)" as if encrypting was the same as signing and verifying was the same as decrypting. I would suggest once again that you stop making these associations: these are 4 distinct operations. While the maths may be the same when using RSA, this doesn't work for DSA, which is only a signature algorithm (so signing/verifying only).
When the client receives the encrypted cryptographic information, it will use the public key in ServerCertificate message to verify(decrypt) and get the plain-text cryptographic information.
The client doesn't receive any encrypted data during the handshake (only signed data).
For a better general understanding, you should start by trying to understand how Diffie-Hellman and its ephemeral variant (for DHE cipher suites) work.
(In practice, I wouldn't focus too much on non-ephemeral DH_RSA/DH_DSS cipher suites. To be honest, I'm not sure whether they're much used. I haven't seen any example of certificate with the necessary DH attributes, and these cipher suites are not in supported lists by OpenSSL or Java 7. DHE/EDH are much more common, and don't require special attributes in the certificate.)
If you use an RSA key exchange algorithm, the client will encrypt the pre-master-key in the client key exchange message; if it's using one of the DH key exchange algorithms, it will send its DH parameters so that client and server can agree on a pre-master-key (in this case, the client will have checked that the server's DH parameters come from the right server by verifying the signature of the server key exchange message sent beforehand). See description of the Client Key Exchange Message:
With this message, the premaster secret is set, either though direct transmission of the RSA-encrypted secret or by the transmission of Diffie-Hellman parameters that will allow each side to agree upon the same premaster secret.
Regarding the other points:
a certificate is only a private-key-signed(encrypted) message that also contains someone else's(e.g., a client ) public key, so a client should use its trusted copy of the server's public key (usually, web browsers include dozens of these certificates in advance), instead of the public key in the server certificate, to verify the server's identity.
Three things happen to verify you're talking to the right server:
- The handshake itself, if successful, guarantees that you're talking to the server that has the private key for the certificate it has presented in the server certificate message. If using RSA key exchange, this is guaranteed by the fact it's the only one that can decipher what the client sent in the client key exchange message (since it's encrypted with the public key); if using an EDH key exchange, this is guaranteed because the server signed its DH parameters in the server key exchange message, verifiable with this public key.
- The fact that you can verify the certificate itself. This is rather independent of how TLS works, but it's usually done using a PKI: the client has a pre-set list of trusted CA certificates, the public keys of which can be used to verify the signature in new certificates it doesn't already know about (such as the server certificate). Verifying that signature allows the client to bind that public key to an identifier (Subject DN and/or alt. name). This gives you the identity of the server to which the client is talking.
- The host name verification: it's not good enough to know that you're talking to someone who's presented you a genuine ID that's valid for them, you also need to check that the name matches the server you intended to connect to.
When I said "The public key in the server certificate isn't used to verify the server's identity itself", I meant that the public key wasn't used to verify points 2 and 3. Point 1 guarantees you that you're talking to the server that has the private key matching the certificate it presented, but it doesn't tell you who this is. Verifying the identity of the server is is up to point 2, so as to be able to bind an identifier to that key/cert.