Why is my JWT bearer authentication recognizing tokens as expired 5 minutes after the token says?

So, what's up? Is there some 5 minute buffer built in that I'm not aware of? What's going on?

What you call "buffer" is actually a built-in feature offered by the JWT bearer middleware (developed by Microsoft) which is known as "clock skew" and was designed to mitigate the effects of clock desynchronization in web farms.

As you figured out, the default value is set to 5 minutes, but it can be changed via JwtBearerOptions.TokenValidationParameters.ClockSkew.