Why can't Let's Encrypt support wildcard certificates?
As a result of an announcement by Let's Encrypt, the information in this answer is now of historical value only. See the information in Emil Stenström's answer.
The IETF-standardized ACME v2 protocol will be available soon, and Let's Encrypt announced on 14 June, 2017 that they will have a new API endpoint against that standard in Jan. 2018. As reported by Emil in his answer, Let's Encrypt announce today (6 June, 2017) that they will be able to use the new API endpoint to issues the much demanded wildcard certificates. Good catch Emil.
After a, too long, exchange in comments, I've decided to collect, and expand, them into an answer. Hopefully this will help. I'm avoiding any discussion of the OV and EV certs because they have a whole different scenario, and the title of the question clearly asks Why can't Let's Encrypt support wildcard certificates?
Top of the list of "why" is that the ACME protocol, which is the foundation of LE's validation, does not yet cover wildcard validation. Without that in place LE simply cannot issue such certs, whether they want to or not. However, there has been discussion on the IETF ACME mailing list, so there may yet be hope for inclusion in the protocol at least. Also, the LE FAQ further explains that wildcard certs are not indefinitely ruled out. Though they may reach that status eventually anyway.
Will Let’s Encrypt issue wildcard certificates?
We currently have no plans to do so, but it is a possibility in the future. Hopefully wildcards aren't necessary for the vast majority of our potential subscribers because it should be easy to get and manage certificates for all subdomains.
Automation of the issuance process is a key principle behind Let's Encrypt. So, anything that cannot be automated will not happen through them. That is one point they have repeatedly stressed in postings on their community forum.
Part of the problem with automating issuance for a wildcard it how it's done. The HTTP validation process is that the cert server tells the requesting agent (some program or script) to create an uniquely named file, with some signed content, on the to-be-validated host. Then the cert server tries to access that file, and verify it's contents. If the file can be retrieved and the contents are correct, it proves that the requesting agent has the technical authority to control that host, and is validated for a cert.
If I want to get a cert for www.example.com the file created will be accessed with the URL of like
http://www.example.com/.well-known/acme-challenge/2BHlJs9Qdk_8m6yRNzqEvwiVejyBsa8zXPpORGMIEz4
The file name is part of a signed block, and is unique to that request. A later request will have a new file name, with new contents, so one cannot pre-load the challenge response in an attempt to bypass validation. Anyway, that's all well and good for a known domain name. What happens when we try to get a cert for *.mysub.example.com? The cert server will attempt to access the URL of
http://*.mysub.example.com/.well-known/acme-challenge/2BHlJs9Qdk_8m6yRNzqEVWiVejyBsa8zXPpORGMIEz4
That's not going to work. It won't even be looked up in the DNS system, not as expected anyway. Trying to look that up in DNS gets you an NXDOMAIN error message under normal circumstances, and when it does return something, it's likely not what's intended. The authentication test can't merely back down one level to mysub.example.com because that isn't covered by the cert requested, and may not be under the control of the requesting agent either. I may be able to create an unlimited number of subdomains for a domain, or subdomain, without the ability to add files to the parent itself.
Adding a wildcard record to the DNS actually works backwards to expectations. Such a record will only match when there is no matching domain in the records. If I create a subdomain of mysub.example.com, with the proper DNS records for it, then *.example.com will not match that. And it carries across record types too. So, a CNAME record for mysub.example.com without an MX record to match will still fail to match a request for an MX record using mysub even though there is a wildcard MX record. RFC 1912 can be interesting, and convoluted, reading. {Here be dragons.}
Since a wildcard in the DNS is counter to what the wildcard in the cert means, things get even more interesting for verification. Looking at the possiblity that I could get several domains registered:
david.jones.namejames.jones.namesarah.jones.namemary.jones.nameangelica.jones.name
I then try to get a wildcard cert for *.jones.name. As validation they request, say 4 domain names in the .jones.name namespace. I can provide any 4 of the 5 available, and meet that test. Now I have a cert for *.jones.name. What is Jared Jones supposed to do when he gets jared.jones.name and I do a MitM attack using my *.jones.name cert?
Due to all of this, they cannot automate verification that the request for a wildcard cert comes from a source that is responsible for ALL and ANY subdomains that might now, or ever, exist in that namespace. They can verify that someone has control over sub.example.com using automated tools, and as long as sub is replaced with something specific that can be validated as well. Once sub is replaced with * the tools for validation don't work. The only way to verify authority over a wildcard domain namespace is to verify ownership of the parent domain AND identity of the requesting party. If I have ownership of the parent domain example.com then I can freely create and control anything as a subdomain, at any level I choose. Note that here "ownership" is distinct from "control", which is what is validated by the ACME protocol.
Most often, outside of LE, a cert is obtained through the host or registrar, so the "identity" is already confirmed, and the ownership is known. If I register with GoDaddy, and host with DigitalOcean, they both "know" who I am, and have a reasonable level of confidence that I own the domain. Even if they don't issue the cert themselves, they can process the request, and verify to the CA that identity and ownership are valid. This allows the CA to issue a wildcard cert for the level requested, even if it is a level or two left of the domain I own, since the entire tree falls under my ownership.
Just looking in the whois for the domain and knowing what the rules are for that namespace is often sufficient to determine ownership. It's a human-centric process, however, and not automatable with any reasonable effort and level of reliability. To complicate things, namespace rules are, at best, inconsistent. For example, for the .name gTLD some 2nd level names are registrable, whilst others are reserved and the 3rd level names are registrable. This also applies to many ccTLDs. And some gTLDs are always 3rd level registration, .pro is an example, i think. How is an automated process to "know" if *.smyther.name is legal rather than *.david.smyther.name?
smith.name is reserved, and people have to register at the third level, so a *.smith.name would be a bad cert to issue. OTOH smyther.name is not reserved, so a *.smyther.name would be a good cert to issue. You can find out by using the whois system. A program would have to parse the output, which is always subject to change, hence unreliable for automation. I know about the .name gTLD, but I have also run across others that have weird, and changeable, rules make automation, maybe not impossible, but impractical. LE is free so they can't do every bell & whistle just because it's nice.
Again, with regard to LE issuing wildcard certs, it comes down to "automation." They 'could' build in conditionals for one case of this, and for another case of that. But where do they stop "tweeking" the rules? The more branches they build into the decision tree, the more points of failure they introduce. Because it's free and automatic, the wildcard has less value. You can get a cert that covers 100 domain names in a single shot (iirc), so why use a wildcard? Their time and effort is better spent on things with more value for the "greater good" than building exceptions and monitoring changing conditions.
Yes, I know there are use cases where a wildcard cert is indispensable. A mobile app that gets a new, random and unique subdomain on every connection couldn't use LE certs. There are limits on how many certs you can request per week, so they couldn't get a new cert every time a client connects. Technically possible, but not going to happen with LE. There are, however, other options for most of such edge cases. One of which is to buy a wildcard cert elsewhere. In many cases involving mobile apps (or even desktop applications), it's also possible to create an internal CA, and add that to the trust database of the app, then create the new short-lived certs, signed internally, as needed.
Let's Encrypt just announced that they will offer wildcard certificates in 2018:
Wildcard certificates will be offered free of charge via our upcoming ACME v2 API endpoint. We will initially only support base domain validation via DNS for wildcard certificates, but may explore additional validation options over time. We encourage people to ask any questions they might have about wildcard certificate support on our community forums.
Source: https://letsencrypt.org/2017/07/06/wildcard-certificates-coming-jan-2018.html
Wildcard support support is now live as of 2018/03/13! Link for steps necessary https://community.letsencrypt.org/t/acme-v2-production-environment-wildcards/55578