how to allow specific crawler from robot text code example
Example 1: seo disallow
User-agent: *
Disallow: /
Example 2: robots txt no index entire site
User-agent: *
Disallow: /
User-agent: *
Disallow: /
User-agent: *
Disallow: /