How does CAPTCHA mitigate DDoS attacks?
BY THE MAGICK POWER OF UNICORNS!!!
Snark aside, CAPTCHA is a very poor solution for D/DoS protection. While it does have some effect, this is minimal, and easily compensated for by the attackers.
CAPTCHA solves the wrong problem for this, and solves it badly.
CAPTCHA does not try to rate-limit the connections; it is not intended to protect the login mechanism from attacks; it is not possible for CAPTCHA to differentiate between attackers and legitimate users.
The only thing it does do - and not very well, at that - is, not surprisingly: Tell Computers and Humans Apart. This might in fact be useful to you, if you were in a Matrix-like reality, where all Computers are the enemy, and all Humans are allies.
In our reality, that is a pointless differentiation. Ostensibly, this might help with preventing scripted attacks - but even if this were true (it's not), there are plenty of ways to bypass that requirement - e.g. CAPTCHA proxies (where you need to solve a CAPTCHA before we show you dem pr0n) and CAPTCHA farms (where you hire some fareasterners to "solve" CAPTCHA for you, at 4$ per 1000 pops).
Moreover, in the case of DDoS - often this attack is mounted by "political movement" - i.e. large masses of humans decide to bring down a certain site. So yeah, CAPTCHA would be irrelevant here.
Besides all that, state-of-the-art CAPTCHA is far behind state-of-the-art OCR. If you want your users to easily decipher those squiggly images - the computer can do this too, pretty well. The best CAPTCHAs were measured at 20% OCR success rate - which effectively means that for every successful request, the attacker would simply need to send 5 requests. Not quite the order of magnitude that would make a substantial difference, to an attacker that is already planning on mounting a DDoS.
Some of my other posts here and on SO regarding this:
- A good summary here on Sec.SE
- Practical non-image based CAPTCHA approaches? on SO
- Has reCaptcha been cracked / hacked / OCR'd / defeated / broken? on SO
TL;DR:
The question is based on a faulty assumption:
How does CAPTCHA defend from DDoS attacks?
It doesn't.
At most it might require a bit more effort from the attacker, but not much.
It is easier/faster to check if a CAPTCHA is correct than to lookup a user in the database (this may include new connections, hashing and more). A server first check the CAPTCHA, if it is correct then process the rest of the login, if not return an error.
It is important to note that making of a CAPTCHA image requires some processing, but this can be done quite effective pre rendered images or even outsourcing to other services (like reCAPTCHA).
A captcha prevents an attacker from performing more database-intensive operations that may cause a DoS via CPU or memory exhaustion. However, this is only the case when the CPU and memory consumption caused by generating the captcha image must be less than that of the normal page request. One way to ensure this is to use an off-site captcha service, such as reCAPTCHA.
Ordinarily, the DoS prevention aspect of the captcha is a by-product of using it as a login security measure, to prevent automated login attempts.