How is a keyvault more secure than standard storage?

Azure Key Vault uses Hardware Security Modules (HSMs) by Thales. Special about HSMs is that they do not give you keys. You create or import a key into an HSM. Later on you give data to the HSM and the HSM is executing cryptographic operations on that data. E.g. encrypting, decrypting, hashing ect. By the way, those hardware devices are really expensive. With Azure Key Vault you are able to use this protection for a small price. That's one benefit using Azure Key Vault.

To your question why this is more secure: Keys are more worth than a single VM that has been compromised. If one of your VMs have been compromised and you would have your encryption keys on that VM, the attacker would have your keys.

If you have only your client ID and secret on your VM, then the attacker has only those credentials but not your keys. And as you know even with those credentials the attacker is not able to get your keys from the HSM. That means the attacker would be able to execute cryptographic operations for a certain amount of time with your credentials. The moment you recognize the attack you invalidate the credentials.

So the difference in a nutshell:

Without HSM: the attacker has your keys and can use those keys as long as they want (and they are valid).

With HSM: the attacker does not have your keys and is only able to execute cryptographic operations with your credentials as long they are valid.