Conditionally disable ASP.NET MVC Controller
Answered here - Prevent ASP.NET Core discovering Controller in separate assembly
This approach doesn't need filters, and hides controller from swagger etc.
The easiest would probably be to implement a custom action filter:
http://www.asp.net/mvc/tutorials/older-versions/controllers-and-routing/understanding-action-filters-cs
You can also conditionally add a route that matches that controller that would result in a 404 being returned.