How can I customize the error response in Web API with .NET Core?

When using a controller with the ApiController attribute applied, ASP.NET Core automatically handles model validation errors by returning a 400 Bad Request with ModelState as the response body. As such, your conditional testing ModelState.IsValid is essentially always false (and therefore not entered) because the only requests that will ever get this far are valid ones.

You could simply remove the ApiController attribute, but that removes a bunch of other beneficial stuff the attributes adds as well. The better option is to use a custom response factory:

services.Configure<ApiBehaviorOptions>(o =>
{
    o.InvalidModelStateResponseFactory = actionContext =>
        new BadRequestObjectResult(actionContext.ModelState);
});

That's essentially what's happening by default, so you'd simply need to change the action provided there accordingly to customize it to your whims.

As Chris analyzed, your issue is caused by Automatic HTTP 400 responses.

For the quick solution, you could suppress this feature by

services.AddMvc()
        .ConfigureApiBehaviorOptions(options => {
            options.SuppressModelStateInvalidFilter = true;
        }).SetCompatibilityVersion(CompatibilityVersion.Version_2_2);

For an efficient way, you could follow the suggestion from Chris, like below:

services.AddMvc()
        .ConfigureApiBehaviorOptions(options => {
            //options.SuppressModelStateInvalidFilter = true;
            options.InvalidModelStateResponseFactory = actionContext =>
            {
                var modelState = actionContext.ModelState.Values;
                return new BadRequestObjectResult(FormatOutput(modelState));
            };
        }).SetCompatibilityVersion(CompatibilityVersion.Version_2_2);

And, there isn't any need to define the code below any more in your action.

if (!ModelState.IsValid)
{
    return BadRequest(FormatOutput(ModelState.Values));
}