example robots.txt
Example 1: seo disallow
User-agent: *
Disallow: /
Example 2: robots txt
# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/
# Group 2
User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml
Example 3: robots txt no index entire site
User-agent: *
Disallow: /