Is demanding a "donation" before disclosing vulnerabilities black hat behavior?

To my understanding, this is no longer in line with responsible 'white hat' behavior. Am I right in this assertion?

White (and grey/black) hat are vague terms. There is no fixed universal definition. By the wikipedia definitions most researchers would be viewed as Grey Hat seeing as its not uncommon to publish if the software publisher refuses to patch.

The easy answer to your question is no. If your aim is to make the world more secure then this clearly does not directly align. There is an indirect argument - that by encouraging financial reward it encourages a healthier relationship between businesses and researchers as well as encouraging more people into the field.

Why do you even ask? The researcher is in no way obliged to disclose to you. Unless you can prove he broke the law in finding the vulnerabilities you have no leverage to force him to. To date you have received several hours worth of work from a (hopefully - otherwise that casts yourselves into a bad light) high skilled individual entirely for free. Either you view it as worth paying him to continue offering services or you don't.

*Seeing the title I would argue this is not Black Hat behavior unless he deliberately exploits the vulnerabilities for his own gain or your own harm (/gives them directly to someone with that intention). If he just refuses to disclose the argument would be between white/grey hat definitions.

The researcher did not create the vulnerability and has not threatened to release or exploit what he has found. If you do not wish to pay for his work then don't and your company is no worse off than it was before he contacted you. In fact he has given you a gift of telling you that there is a vulnerability which you can find yourselves or through another contractor.

So no, he has not done anything unethical.

I'm a bug hunter and I have no idea why everybody here thinks it's perfectly fine of him to attack your website without permission, determine a bounty amount himself, and threaten to hold back potentially dangerous flaws because he doesn't get the money he wants. Why didn't he ask about your policy beforehand? You never claimed to run a bug bounty program, or to be able to pay anyone for anything in the first place.

To clarify again, OP did not sign up for the Open Bug Bounty project. The project offers to be an intermediary between researchers and websites that don't run a bounty program. Also, they explicitly mention that you are not obliged to pay anything, and the researcher should be well aware of that if he read the guidelines.

A website owner can express a gratitude to the researcher in a way s/he considers the most appropriate and proportional to the researcher's efforts and help. We encourage website owners to say at least a “thank you” to the researcher or write a recommendation in the researcher’s profile. There is, however, absolutely no obligation or duty to express a gratitude.

(Emphasis my own)

If he expects a monetary reward, he should be searching for bugs at companies that actually run a bounty program. There are plenty of reputable programs paying high rewards.

The researcher then sent a follow up email saying that he has found more vulnerabilities, but because we didn't make a donation, he will keep those vulnerabilities for himself.

If he already found them and it's not much effort to put them in a list and let you know, then yes, I find it unethical to hold the bugs back.¹ You didn't ask him to do work for you. He could have inquired if you pay for bugs beforehand. And he didn't so much offer you his expertise for a donation - from your description it sound more like a mild threat that if you don't pay him, your website is in danger.

Take that incident as a hint that you need to invest more in your security. Consider setting up a real bug bounty program with small bounties or hiring a professional penetration tester. But don't let him extort money from you if you never promised any.

¹It's not that he should do free work for you. It's the fact that he mentions that there is something he won't tell you unless you pay him a particular amount that makes it unethical, especially if an exploitation of these bugs could threaten the future of your company.