Why don't VPN services use TLS?
There are a few things to undestand.
The first is that most VPN tools were originally designed to provide private connectivity over networks that were insecure, possiblly natted or firewalled but not actively hostile to VPNs.
The second is that traditional TLS is a wrapper over TCP. TCP is a poor choice for VPNs because it suffers from "head of line blocking". If one packet gets lost then all the data behind it is blocked until it is successfully retransmitted. This used to cause big problems when running VPNs over TCP, it's less of an issue now that we have TCP fast retransmission but it's still less than ideal.
The third is that deep packet inspection is a fairly new thing. Networks that will pass legitimate TLS traffic unmolested but that will not pass other traffic on port 443 are still the exception not the rule.
While Openvpn does have a TCP option it is primerally designed to run over UDP. It uses TLS for key exchange but encrypts the actual network packets using a system designed explicitly for that purpose.
https://openvpn.net/index.php/open-source/documentation/security-overview.html
Someone should say something about DTLS (TLS over UDP) too
DTLS is a pretty new protcol. I see no reason why you couldn't/shouldn't build a VPN soloution on top of it but I guess the VPN software vendors aren't especially in the mood to redesign their software.
In any case it wouldn't help with the scenario described in the question.
Short version:
- VPN protocols need to provide encapsulation that TLS doesn't provide
- Of course, it's possible to tunnel VPN over TLS to get the same effect!
Long version:
The name Virtual Private Network suggests the answer to your question. A true VPN protocol emulates a Network, which implies the ability to route multiple traffic types and ports simultaneously over a single channel. That's why with a VPN protocol like IPSec you can end up with, for example, TCP packets encapsulated within ESP packets encapsulated within other TCP packets. (See also What's the difference between SSH and IPsec?)
By default TLS doesn't provide this capability. TLS takes bytes, encrypts them, and transfers them to the recipient who can decrypt it. That's it; everything else that's to be done has to be done by the application. Under HTTPS, a web server and client exchange HTTP data bytes over a TLS channel. Under IMAPS, a mail server and client exchange IMAP data bytes over a TLS channel.
In order to have generic "Network" connectivity over TLS, you need a client and server able to do that encapsulation and decapsulation on either end. You don't get it for free.
That's not to say you can't have a VPN that runs over TLS. There are many products that do it; even mainline vendors like Checkpoint support both IPSec and TLS-based VPNs. In some cases the TLS is simply encapsulating IPSec datagrams, so the actual "N" is via IPSec but the TLS gets it across the Internet.
Note that in some cases a sophisticated adversary can infer when a TLS connection is being used for VPN based on the pattern of communication. There's more to Operational Security than encrypting bits!
Many VPN protocols do use TLS. In particular, almost all of the modern client-server VPNs (e.g. used to connect a laptop remotely to a corporate network) support TLS as a primary or fallback transport.
I'm a contributor to the well-known open-source client for TLS/SSL based VPNs: openconnect. I reverse-engineered the GlobalProtect VPN protocol (TLS+ESP) almost entirely on my own. I've studied enough VPN protocols in detail to notice some pretty clear patterns and design tradeoffs.
Advantages of TLS for VPN authentication and transport:
- Robustness. TLS-based VPN traffic is indistinguishable from "normal" HTTPS traffic in terms of its packet structure and encrypted contents — though timing and size of packets, and duration of connections, hint that it's carrying something other than "normal" browser traffic. The universality of HTTPS means that TLS-based VPNs can get through almost all firewalls and middleboxes; only really determined censors who do traffic analysis, or blacklist VPN endpoints by DNS or IP, can block it.
- Strong security guarantees: modern TLSv1.2 (soon, 1.3) is a very well-designed, carefully analyzed, and gradually-matured protocol for secure communications. Many libraries wrap it in a way that makes it fairly hard to misuse. If you use a TLS-based VPN, you can expect that your connection to the gateway uses modern ciphers (no ancient 1DES, for example) and x509 public-key certificate verification of the gateway, limiting man-in-the-middle attacks. It's possible to screw up the implementation of TLS, badly breaking its security, but relatively difficult due to the standardization of high-quality libraries which do most of the tricky bits. It would be much, much easier to write an insecure VPN based on a "custom" protocol.
Disadvantages of TLS for VPN transport, compared to ESP (the normal transport layer of IPSEC-based VPNs):
- TLS normally runs over TCP, which performs poorly as a tunneling transport with congested or lossy underlying network conditions… especially in the case of TCP-over-TCP.
- Fortunately, DTLS ("datagram TLS") can alleviate this; it's used most prominently in the AnyConnect/openconnect VPN protocol, as well as in real-time communications protocols like WebRTC.
- Most modern operating systems have optimized in-kernel support for IPSEC and ESP. This makes ESP really efficient and low-overhead for gateways and firewalls and end-client systems, when implemented properly. TLS is typically implemented in userspace, and combined with the complexity of the underlying TCP stream, this makes it less efficient.
- (A similarly efficient in-kernel implementation of DTLS should be possible, but I'm not aware of any in a mainstream operating system.)
… as far as using TLS for the authentication/setup of a VPN? I basically see no real disadvantages there, other than the lack of standardized protocol for it.
TLS is more flexible, widely-implemented, and easier to debug than ISAKMP (the authentication/key exchange component of IPSEC). Several modern corporate VPN protocols, including GlobalProtect, use TLS for authentication and key exchange but then ESP for data transport (though they can fall back to TLS if needed). This is a pretty effective and efficient combination, in my experience.