My ISP uses deep packet inspection; what can they observe?
Deep Packet Inspection, also known as complete packet inspection, simply means they are analyzing all of your traffic as opposed to just grabbing connection information such as what IP's you are connecting to, what port number, what protocol and possibly a few other details about the network connection.
This is normally discussed in contrast to the gathering of NetFlow information which mainly collects the information listed above.
Deep packet inspection gives your provider a lot of information about your connections and habits of Internet usage. In some cases, the full content of things like SMTP e-mails will be captured.
HTTPS does encrypt the connections but your browser has to make DNS requests which are sent primarily via UDP so that data will be collected as will any unencrypted links or unencrypted cookies sent incorrectly without https. These additional bits which will be collected may be very telling about what type of content you are looking at.
The larger concern for most people is about data aggregation, by collecting this information a data scientist could create a fingerprint for your Internet usage and later associate with past activities or activities from other locations (when you are at work or are on vacation). Likewise, your service provider may choose to sell this to any number of organizations (possibly including criminal organizations) where it could then be used against you in ways. In many countries, people have an expectation that their communications are considered to be private and collecting this data very much goes against that privacy expectation.
Another interesting aspect of this is in the cases like the US where this data may soon be sold it allows International communications sent to people, or servers, in the US be sold as well. Likewise, this could potentially allow every agency from local law-enforcement, military, tax authorities, immigration authorities, politicians, etc. a way to bypass long-standing laws which have prevented them from accessing this type of information, or important informational subsets within this data otherwise.
A slightly different concern when this data can be sold is competitive intelligence / corporate espionage. In the scenario where a company does a lot of research-intensive work at their headquarters located in some small geographical location (think of pharmaceuticals or a defense contractor) selling that data makes it possible for anyone to buy all of the traffic from the local ISP where most of those researchers live and analyze what they are looking for when at home, possibly even directly from the ISP hosting the traffic for their corporate headquarters. If other countries aren't selling similar data it gives foreign companies and companies wise enough to try and buy this data a huge technical advantage. Likewise, it would also allow foreign governments to buy ISP traffic which includes the data from US (or other government) Officials homes.
Imagine companies monitoring their employee's behavior at home or on their mobile devices.
This will likely have a chilling effect on activists and whistle-blowers as well.
Likewise, if credit cards or PII are sent in the clear to a poorly secured remote site your ISP's data set now has a potential PCI or PII regulatory issue on their hands. So this amplifies data-leakage problems of all types by making additional copies of the data leaked.
With the examples I've just mentioned above, and there are hundreds of others, it should be easy to see why this type of data collection has a different level of importance to it than just metadata or basic connection information. Even if your ISP never sells this data they are collecting quite an interesting dataset.
It's a security issue that definitely has a lot of potential long-term security implications.
Trey Blalock's answer describes percisely what deep packet inspection (DPI) is. But I'd like to add three things to hopefully answer your specific questions:
- There is a technique of DPI that does decrypt your data, called SSL interception, although it is more common in enterprise situations and only possible if the ISP (or any other interceptor) has the ability to install a certificate on your machine. So unless the ISP had some way of doing that (technician etc.) this is probably off the table.
- HTTPS would prevent the ISP from being able to read data. Of course, This is only true for services that use HTTPS (which unfortunately is not all of them). Also you need to consider that the ISP can read metadata whether the connection is encrypted or not.
- A VPN would protect you against DPI performed by the ISP (not by the VPN provider). This is thanks to the fact that VPNs use an encrypted tunnel to connect you to the 'exit node'. This encrypts all of your traffic, and all of the metadata will show packets leaving your computer and going to the VPN server (thus not disclosing the actual server you are accessing).
As stated by Trey, DPI can see the entire content of your network traffic. All of it. If it is plain text, then they see everything that you do.
To add on to Miao's answer:
Things DPI can see, even when you use HTTPS:
- DNS information, e.g. https://catvideos.com/tigers - they will see https://catvideos.com
- IP address connectivity. So, even you you HTTPS to that site with cat videos, they can see that you connected to that cat video site and downloaded 500 GB of data. They don't know what data, but they know the DNS name, and the IP address, and amount of data to that site, and to every site.
- Ads. Many/most Ad networks do not use HTTPS, so that data is not always encrypted. This can result in a "mixed encryption" or similar warning from a browser.
- other data: Many sites using HTTPS for login will then drop the encryption for everything else.
- graphics: many sites won't encrypt things like their logo or various graphic or video files. They may encrypt your login and search, but not results.
- other non-HTTPS traffic like UDP, mail, SNMP, ftp, telnet, updates to some software might not use HTTPS, etc.
With a VPN, they will still see 100% of the data. However, other than the connection to the VPN provider, they'll only see encrypted data. They will know that you downloaded 800GB from VPNco.com, but will know nothing of the data inside. Even things that are not encrypted via protocol will get encrypted since a lower level is encrypting. Now, the VPNco.com will then see your data.
With the (potential) change in US law about ISP and data privacy, combined with the (potential) loss of net neutrality, ISP's might be able to not only see 100% of your data, they could modify that data, slow or block sites they want, and might be able to sell any/all of your data to a 3rd party (as Trey states).
I'm not covering MITM (like Miao states in #1 above), since you stated ISP, I assume that you are talking about a home system and a DSL or cable modem.
https://stackoverflow.com/questions/499591/are-https-urls-encrypted