What's the point of the X-Requested-With header?
A good reason is for security - this can prevent CSRF attacks because this header cannot be added to the AJAX request cross domain without the consent of the server via CORS.
Only the following headers are allowed cross domain:
- Accept
- Accept-Language
- Content-Language
- Last-Event-ID
- Content-Type
any others cause a "pre-flight" request to be issued in CORS supported browsers.
Without CORS it is not possible to add X-Requested-With to a cross domain XHR request.
If the server is checking that this header is present, it knows that the request didn't initiate from an attacker's domain attempting to make a request on behalf of the user with JavaScript. This also checks that the request wasn't POSTed from a regular HTML form, of which it is harder to verify it is not cross domain without the use of tokens. (However, checking the Origin header could be an option in supported browsers, although you will leave old browsers vulnerable.)
New Flash bypass discovered
You may wish to combine this with a token, because Flash running on Safari on OSX can set this header if there's a redirect step. It appears it also worked on Chrome, but is now remediated. More details here including different versions affected.
OWASP Recommend combining this with an Origin and Referer check:
This defense technique is specifically discussed in section 4.3 of Robust Defenses for Cross-Site Request Forgery. However, bypasses of this defense using Flash were documented as early as 2008 and again as recently as 2015 by Mathias Karlsson to exploit a CSRF flaw in Vimeo. But, we believe that the Flash attack can't spoof the Origin or Referer headers so by checking both of them we believe this combination of checks should prevent Flash bypass CSRF attacks. (NOTE: If anyone can confirm or refute this belief, please let us know so we can update this article)
However, for the reasons already discussed checking Origin can be tricky.
Update
Written a more in depth blog post on CORS, CSRF and X-Requested-With here.
Make sure you read SilverlightFox's answer. It highlights a more important reason.
The reason is mostly that if you know the source of a request you may want to customize it a little bit.
For instance lets say you have a website which has many recipes. And you use a custom jQuery framework to slide recipes into a container based on a link they click.
The link may be www.example.com/recipe/apple_pie
Now normally that returns a full page, header, footer, recipe content and ads. But if someone is browsing your website some of those parts are already loaded. So you can use an AJAX to get the recipe the user has selected but to save time and bandwidth don't load the header/footer/ads.
Now you can just write a secondary endpoint for the data like www.example.com/recipe_only/apple_pie but that's harder to maintain and share to other people.
But it's easier to just detect that it is an ajax request making the request and then returning only a part of the data. That way the user wastes less bandwidth and the site appears more responsive.
The frameworks just add the header because some may find it useful to keep track of which requests are ajax and which are not. But it's entirely dependent on the developer to use such techniques.
It's actually kind of similar to the Accept-Language header. A browser can request a website please show me a Russian version of this website without having to insert /ru/ or similar in the URL.
Some frameworks are using this header to detect xhr requests e.g. grails spring security is using this header to identify xhr request and give either a json response or html response as response.
Most Ajax libraries (Prototype, JQuery, and Dojo as of v2.1) include an X-Requested-With header that indicates that the request was made by XMLHttpRequest instead of being triggered by clicking a regular hyperlink or form submit button.
Source: http://grails-plugins.github.io/grails-spring-security-core/guide/helperClasses.html