what is robots.txt file code example
Example 1: robots.txt disallow
User-agent: *
Disallow: /
Example 2: robots txt
# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/
# Group 2
User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml