How to bypass authentication middleware when not needed in ASP.NET Core

If you add Authorization to your middleware pipeline this will be the default for all calls to your API. Therefore all calls will act as though they have the [Authorize] attribute applied.

This is normally desirable as it means by default your application is secure and you can't accidently forget an [Authorize] attribute. I'd recommend keeping it like this and simply add the [AllowAnonymous] tag to the controllers or controller actions you want to be public.

If you want to be explicit at all times you simply remove app.UseAuthentication(); You will still be able to use [Authorize] which will trigger your middleware as you have added the service for use. But it will not automatically trigger for all calls.

Additional:

In order to use authorization without having to specify the scheme per call you can set your scheme as a default authorization policy.

services.AddAuthorization(options =>
{
    var defaultAuthorizationPolicyBuilder = new AuthorizationPolicyBuilder(
        CustomAuthenticationSchemeOptions.SchemeName);

    defaultAuthorizationPolicyBuilder = 
        defaultAuthorizationPolicyBuilder.RequireAuthenticatedUser();

    options.DefaultPolicy = defaultAuthorizationPolicyBuilder.Build();
});

To extend what others have said there are subtle differences between Authentication and Authrorization. Authentication says who a user is, Authorization say what they are allowed to do. All the above simply says is... provided I know who a user is (Is Authenticated) they are allowed to use my actions (Is Authorized). So your default authorization policy is effectively if a user is successfully authenticated.

I think you are confusing authentication and authorization. From the docs,

Authentication is a process in which a user provides credentials that are then compared to those stored in an operating system, database, app or resource. If they match, users authenticate successfully, and can then perform actions that they're authorized for, during an authorization process. The authorization refers to the process that determines what a user is allowed to do.

What you have created is an authentication handler. Authentication runs regardless of whether there is an authorize attribute, as the purpose of authentication is not only authorization. It is also identification. You should setup authentication in a way that does not cause db load on every request, such as by issuing in a JSON Web Token or an Application Cookie at a selected endpoint such as /account/login. Once you have authentication setup in this manner, you can setup a custom authorization handler or authorization attribute that checks the claims through HttpContext.User.Claims to see whether the User is authorized to access the resource, (and even perform light db activity for this matter, such as looking up user roles.)

This guide outlines how to setup authorization handlers, but you can start by reading up on authorization first.