Does default ModSecurity protect enough against XSS?

You should review the XSS Street-Fight (with ModSecurity) Blackhat preso

It outlines the following ModSecurity mitigation strategies for XSS:

  1. Input Validation (Whitelist/Blacklist Filtering)
  2. Generic Attack Payload Detection
  3. Identifying Improper Output Handling Flaws (Dynamic Taint Propagation)
  4. Application Response Profiling (Monitoring the number of scripts/iFrames)
  5. JavaScript Sandbox Injection (ModSecurity’s Content Injection Capabilities)

I was going to suggest that you check out Ryan Barnett's work, but he already answered!

Data validation is not enough to prevent XSS, even if it's a pure whitelist.

The identification of improper output handling must occur on all outputs. They might be able to be fixed contextually with ModSecurity, but certainly this is the wrong place in the architecture to do it -- since if anything changes with regards to that content -- the encoding/escaping would suddenly become useless. Web content has a way of changing a lot.

The correct answer is to monitor output escaping issues with ModSecurity -- and actually remediate XSS issues elsewhere.

One of the best approaches I've heard lately is to Stop Building HTML on the Server. In particular, this would kill two birds with one stone: you could solve the problems with AJAX (e.g. DOM-based XSS) as well as stored and reflected XSS issues.

However, I strongly encourage you to check out the Encoding libraries in OWASP ESAPI and OWASP ESAPI JS. The best remediation advice comes from contextual output encoding. Remediating XSS is a lot of work, but it's worth to think about these problems as longer-lasting and with serious impact if we don't fix them now.