What kinds of security vulnerabilities does providing DNSSEC expose?
Solution 1:
DNSSEC has some risks, but they are not directly related to reflection or amplification. The EDNS0 message size expansion is a red herring in this case. Let me explain.
Any exchange of packets that does not depend on a previous proof of identity is subject to abuse by DDoS attackers who can use that unauthenticated packet exchange as a reflector, and perhaps also as an amplifier. For example, ICMP (the protocol behind "ping") can be abused in this way. As can the TCP SYN packet, which solicits up to 40 SYN-ACK packets even if the SYN was spoofed to come from some victim who doesn't want those SYN-ACK packets. And of course, all UDP services are vulnerable to this attack, including NTP, SSDP, uPNP, and as noted by other responses here, also including DNS.
Most intrusion detection, intrusion prevention, and load balancer appliances are bottlenecks, unable to keep up with "line rate" traffic. Also many routers can't run at line rate, and some switches. These bottlenecks, by being the smallest thing "in the path", and smaller than the links themselves, are the actual target of congestion-based DDoS attacks. If you can keep somebody's firewall busy with attack traffic, then good traffic won't get through, even if the links aren't full. And what slows down a firewall isn't the total number of bits per second (which can be increased by using larger messages, and EDNS0 and DNSSEC will do), but rather the total number of packets per second.
There's a lot of urban legend out there about how DNSSEC makes DDoS worse because of DNSSEC's larger message size, and while this makes intuitive sense and "sounds good", it is simply false. But if there were a shred of truth to this legend, the real answer would still lay elsewhere-- [because DNSSEC always uses EDNS0, but EDNS0 can be used without DNSSEC], and many normal non-DNSSEC responses are as large as a DNSSEC response would be. Consider the TXT records used to represent SPF policies or DKIM keys. Or just any large set of address or MX records. In short, no attack requires DNSSEC, and thus any focus on DNSSEC as a DDoS risk is misspent energy.
DNSSEC does have risks! It's hard to use, and harder to use correctly. Often it requires a new work flow for zone data changes, registrar management, installation of new server instances. All of that has to be tested and documented, and whenever something breaks that's related to DNS, the DNSSEC technology must be investigated as a possible cause. And the end result if you do everything right will be that, as a zone signer, your own online content and systems will be more fragile to your customers. As a far-end server operator, the result will be, that everyone else's content and systems will be more fragile to you. These risks are often seen to outweigh the benefits, since the only benefit is to protect DNS data from in-flight modification or substitution. That attack is so rare as to not be worth all this effort. We all hope DNSSEC becomes ubiquitous some day, because of the new applications it will enable. But the truth is that today, DNSSEC is all cost, no benefit, and with high risks.
So if you don't want to use DNSSEC, that's your prerogative, but don't let anyone confuse you that DNSSEC's problem is its role as a DDoS amplifier. DNSSEC has no necessary role as a DDoS amplifier; there are other cheaper better ways to use DNS as a DDoS amplifier. If you don't want to use DNSSEC, let it be because you have not yet drunk the Kool Aid and you want to be a last-mover (later) not a first-mover (now).
DNS content servers, sometimes called "authority servers", must be prevented from being abused as DNS reflecting amplifiers, because DNS uses UDP, and because UDP is abusable by spoofed-source packets. The way to secure your DNS content server against this kind of abuse is not to block UDP, nor to force TCP (using the TC=1 trick), nor to block the ANY query, nor to opt out of DNSSEC. None of those things will help you. You need DNS Response Rate Limiting (DNS RRL), a completely free technology which is now present in several open source name servers including BIND, Knot, and NSD. You can't fix the DNS reflection problem with your firewall, because only a content-aware middlebox such as the DNS server itself (with RRL added) knows enough about the request to be able to accurately guess what's an attack and what's not. I want to emphasize, again: DNS RRL is free, and every authority server should run it.
In closing, I want to expose my biases. I wrote most of BIND8, I invented EDNS0, and I co-invented DNS RRL. I've been working on DNS since 1988 as a 20-something, and I am now grumpy 50-something, with less and less patience for half-baked solutions to misunderstood problems. Please accept my apologies if this message sounds too much like "hey you kids, get offa my lawn!"
Solution 2:
I know of two specific vulnerabilities. There is the reflection/amplification mentioned by Håkan. And there is the possibility of zone enumeration.
Reflection / amplification
Reflection means attacks in which requests with a spoofed source IP are sent to a DNS server. The host being spoofed is the primary victim of the attack. The DNS server will unknowingly send the reply to a host which never asked for it.
Amplification refers to any reflection attack in which the reflected response consists of more bytes or more packets than the original request. Before DNSSEC+EDNS0 amplification in this way would only allow up to 512 bytes to be sent. With DNSSEC+EDNS0 it is possible for 4096 bytes to be sent, which typically spans 3-4 packets.
There are possible mitigations for these attacks, but I don't know of any DNS server implementing them.
When the client IP has not been confirmed, the DNS server can send a truncated response to force the client to switch to TCP. The truncated response can be as short as the request (or shorter if the client uses EDNS0 and the response does not) which eliminates the amplification.
Any client IP which completes a TCP handshake and send a DNS request on the connection can be temporarily whitelisted. Once whitelisted that IP gets to send UDP queries and receive UDP responses up to 512 bytes (4096 bytes if using EDNS0). If a UDP response triggers an ICMP error message, the IP is removed from the whitelist again.
The method can also be reversed using a blacklist, which just means that client IPs are allowed to query over UDP by default but any ICMP error message cause the IP to be blacklisted needing a TCP query to get off the blacklist.
A bitmap covering all relevant IPv4 addresses could be stored in 444MB of memory. IPv6 addresses would have to be stored in some other way.
Zone enumeration
Whether zone enumeration is a vulnerability in the first place is subject of debate. But if you don't want all names in your zone to be public knowledge, you would likely consider it a vulnerability. Zone enumeration can mostly be mitigated through the use of NSEC3 records.
The problem which still persists even when using NSEC3 is that an attacker can find the hash of each label by simply querying for random names. Once the attacker has all the hashes an off-line brute force attack can be performed on those hashes.
A proper defense against zone enumeration would require an attacker to perform a query to the authoritative server for every guess. However no such defense exists in DNSSEC.
Solution 3:
The thing that comes to mind is not actually DNSSEC specific but rather about the EDNS0 extension, which DNSSEC relies on.
EDNS0 allows for larger UDP payloads and larger UDP payloads can allow for worse traffic reflection/amplification attacks.
I don't know what the percentage of validating resolvers is but popular nameserver software seems be shipped with validation on by default and one will easily find some notable service providers that are open about them doing validation, such as Comcast and the Google public resolvers.
Based on this, I would think that the percentage of validating resolvers is probably in significantly better shape than the percentage of signed zones.