How Can I Encourage Google to Read New robots.txt File?
Solution 1:
In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.
Go to Health -> Fetch as Google [1] and have it fetch /robots.txt
That will re-download the file and google will also re-parse the file.
[1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.
Solution 2:
I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:
- first correct your robots.txt to allow the correct pages, then
- upload a sitemap.xml with your pages
as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.