Precedence of X-Robots-Tag header vs robots meta tag
In my recent experience, when Google sees mixed-messages it prefers positive action by default - ie - it favours indexing - meanwhile will flag the issue as a critical error/warning in your webmaster tools console if you have one.
see your site's status in google here: https://www.google.com/webmasters/
see you site's status in bing here: http://www.bing.com/toolbox/webmaster (note that yahoo search is now powered by bing)
Google takes this positive-by-default action because lots of site owners unwittingly have a dodgy cms semi-blocking robots and we know how google loves to accumulate as much data as it can - any excuse!
if the technical settings are erroneous they're liable to be totally disregarded, and we know how search engines index and follow by default when no settings are specified.
I am not sure if a definitive answer can be given to the question, as the behavior may be implementation-dependent (on the robot side).
However, I think there is reasonable evidence that X-Robots-Tag
will take precedence over <meta name="robots" ...
. See :
One significant difference between the X-Robots-Tag
and the robots
meta directive is:
X-Robots-Tag
is part of the HTTP protocol header.<meta name="robots" ...
is part of the HTML document header.
Therefore the the X-Robots-Tag
belongs to HTTP protocol layer, while <meta name="robots" ...
belongs to the HTML protocol layer.
As they belong to a different protocol layer, they will not be parsed simultaneously by the (robot) client getting the page: The HTTP layer will be parsed first, and the HTML in a later step.
(Also, it should be noted that X-Robots-Tag
and <meta name="robots" ...
are not suppported by all robots. Google and Yahoo/Bing suppport both, but according to this some support only <meta name="robots" ...
, others support neither.)
Summary :
- if supported by the robot,
X-Robots-Tag
will be processed first ; restrictions (noindex, nofollow) apply (and<meta name="robots" ...
is ignored). - else,
<meta name="robots" ...
directive applies.
Just an update to Dan's experience, I also have both the
Header set X-Robots-Tag "noindex, nofollow"
and
<meta name="robots" content="index, follow" />
on my one of my Wordpress sites, and a check in Google Search Console confirmed that the noindex in X-Robots-Tag is taking precedence as the pages have been crawled and but aren't indexed. So the logic in the correct answer is indeed, correct.