Unable to map route for robots.txt in asp.net mvc

Creating a route ending with a file extension is not allowed by default in ASP.NET MVC. To get around this security restriction, you need to add the following to the Web.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- ...Omitted -->
  <system.webServer>
    <!-- ...Omitted -->
    <handlers>
      <!-- ...Omitted -->
      <add name="RobotsText"
           path="robots.txt"
           verb="GET"
           type="System.Web.Handlers.TransferRequestHandler"
           preCondition="integratedMode,runtimeVersionv4.0" />
    </handlers>
  </system.webServer>
</configuration>

In Asp.net Core you can simply add your robots.txt file to the wwwroot directory.