How to set robots.txt globally in nginx for all virtual hosts

Solution 1:

You can set the contents of the robots.txt file directly in the nginx config:

location = /robots.txt { return 200 "User-agent: *\nDisallow: /\n"; }

It is also possible to add the correct Content-Type:

location = /robots.txt {
   add_header Content-Type text/plain;
   return 200 "User-agent: *\nDisallow: /\n";
}

Solution 2:

Are there other rules that are defined? Maybe common.conf or another conf file in included which is over-riding your config. One of the following should definitely work.

location /robots.txt { alias /home/www/html/robots.txt; }
location /robots.txt { root /home/www/html/;  }

1. Nginx runs all "regexp" locations in order of their appearance. If any "regexp" location succeeds, Nginx will use this first match. If no "regexp" location succeeded, Nginx uses the ordinary location found on the previous step.
2. "regexp" locations have precedence over "prefix" locations

---

Solution 3:

location cannot be used inside http block. nginx does not have global aliases (i.e., aliases that can be defined for all vhosts). Save your global definations in a folder and include those.

server {
  listen 80;
  root /var/www/html;
  include /etc/nginx/global.d/*.conf;
}