robots txt file disallow all but allow selected code example Example: robots txt disallow all User-Agent: * Disallow: /