Robots.txt file in MVC.NET 4

do robots crawl the controllers which has [Authorization] attribute like Administration

If they find a link to it, they are likely to try and crawl it, but they will fail just like anyone with a web browser that does not log in. Robots have no special ability to access your website differently than a standard browser.

Note that robots that conform to the Robots Exclusion Standard crawl the exact URL

http://mydomain/robots.txt

You can create a response for that URL however you like. One approach is certainly to have a controller that handles that request. You can also just add a text file with the same content you would have returned from the controller, e.g.

User-agent: *
Disallow: /Administration/
Disallow: /Account/

to the root folder of your project and make sure it is marked as content so that it is deployed to the website.

Adding this robots.txt entry will prevent conforming robots from attempting to browse controllers that require authentication (and lighten the load on your website slightly), but without the robots file they will just try the URL and fail.

This simple piece of code worked for my asp net core 3.1 site:

    [Route("/robots.txt")]
    public ContentResult RobotsTxt()
    {
        var sb = new StringBuilder();
        sb.AppendLine("User-agent: *")
            .AppendLine("Disallow:")
            .Append("sitemap: ")
            .Append(this.Request.Scheme)
            .Append("://")
            .Append(this.Request.Host)
            .AppendLine("/sitemap.xml");

        return this.Content(sb.ToString(), "text/plain", Encoding.UTF8);
    }