Using AddAzureKeyVault makes my application 10 seconds slower

Yes, configure the AzureServiceTokenProvider explicitly to use the az cli for authentication. You can do this by setting an environment variable named AzureServicesAuthConnectionString.

Bash:

export AzureServicesAuthConnectionString="RunAs=Developer; DeveloperTool=AzureCli"

PowerShell:

$Env:AzureServicesAuthConnectionString = "RunAs=Developer; DeveloperTool=AzureCli"

Note that the environment variable needs to be set in whatever session you're running your app.

Explenation

The root of the problem is hinted at in MS docs on authentication, which state, "By default, AzureServiceTokenProvider uses multiple methods to retrieve a token."

Of the multiple methods used, az cli authentication is a ways down the list. So the AzureServiceTokenProvider takes some time to try other auth methods higher in the pecking order before finally using the az cli as the token source. Setting the connection string in the environment variable removes the time you spend waiting for other auth methods to fail.

This solution is preferable to hardcoding a clientId and clientSecret not only for convenience, but also because az cli auth doesn't require hardcoding your clientSecret or storing it in plaintext.