Why is end-to-end encryption still not default in mails?

It's not that hard, why isn't it standard for years?

Because that would not have solved the problem that PGP is trying to solve.

PGP is an end to end encryption, so if there is any way for the SMTP server to subvert the encryption, then the scheme fails.

In the case of the scheme you proposed, suppose Alice ([email protected]) wants to send a private message to Bob ([email protected]). Using the scheme you proposed, Alice's mail client or Alice's SMTP server fetches Bob's public key by making TLS connection to dave.com. This is fine as long as dave.com are honest and actually return Bob's public key. But dave.com could have been configured by dave.com's operator to return a forged public key created by Eve, or Eve could hack into dave.com and set this up. Now Alice's mail client/mail server would happily accept Eve's certificate, thinking the public key is Bob's. In this model, the operator of dave.com can intercept any of Bob's emails.

Now, as long as dave.com is honest, this still protects against third party spoofing. Why don't we do this anyway if this protects at least against third party snooping? Mainly because SMTPS also provides the same level of protection, while also being much simpler. If MITM by the mail server operator is not your concern, you can already very well secure your emails by ensuring you both uses SMTPS.

Note that the difficulty of end to end encryption isn't about fetching public keys. Most email client that supports PGP also supports automatically fetching public keys from LDAP or HPKP. The difficulty of end to end encryption is verifying the public keys.

There is no known method of verifying public keys that is fully transparent to the users and fully secure. Web of Trust or Certificate Authority model comes closest, but Web of Trust comes with a lot of caveats and Certificate Authority model relies on a third party to do the verification.

integrating PGP into SMTP.

PGP is a container format for data (like mails but not restricted to mails), which adds encryption and/or signature to the data. SMTP is a transport protocol.

You don't integrate container formats into transport protocols. This would be the same as saying that you should integrate Office (container for text, images...) with SMTP (transport) to send an office document to somebody.

PGP is also used outside of SMTP, because it is just a container. And SMTP is also used to transport things different from PGP containers, because it is just a transport protocol.

If you instead ask about integrating end-to-end encryption like PGP or S/MIME into SMTP it will not work either, because SMTP is hop-by-hop delivery and not end-to-end. Apart from that SMTP does not even cover the last hop, i.e. the delivery from the last mail server to the client. This is done with protocols like POP or IMAP.

Lea requests the server of Luke's domain jedi.com to tell her the public key of [email protected] ...

That's what you have key servers or other kinds of central directories for. But how does Lea know that this is actually Luke's key and not the key of someone claiming to be Luke? Thus you need to have some trust propagation, for example in the form of a web of trust (PGP) or a more centralized structure (S/MIME) or by trusting everything in a specific central directory.

Thus the task is not to integrate PGP with SMTP but to have better support for PGP in the mail clients, so that they automatically fetch the PGP keys of the recipients. But of course there must be first a verifiable PGP key for the recipient somewhere on a key server or other directory, so the other task is to make key creating, publishing and managing keys (renewing, revoking...) easier. These are all things outside of the mail delivery (SMTP) itself.

Encryption is already in place during mail transit (STARTTLS in SMTP), but not sophisticated enough to protect against MITM.

I believe PGP is more of an end-user experience between email clients, which is helpful if you don't have full trust of the servers involved.

(PGP is sometimes susceptible to MITM to the less-than-careful user, however, like in SSH, if you check for the correct key signature then that problem is solved)

However, in the case of cloud-based email services like Gmail, they would have to be available to the server anyway for a good user experience, so PGP would just get in the way.

Hopefully one day we'll get an MITM proof encryption in SMTP, but it is less of an issue there because mail servers are on controlled networks.