Reasonable to exclude a specific professor from a thesis defense for personal reasons?

This is something to talk over with your advisor. It is a reasonable request, but it might not be possible to go along with it. But if your advisor knows the whole story they can advise you on how to proceed, or even, if necessary, assure that the other person isn't included.

But, assuming that your advisor is present at the defense, perhaps one of the examiners, then they can intervene if there is anything improper going on.

If the advisor isn't present, it is a bigger problem, and you need to work harder to achieve the other person's attendance. Sadly, there may not be any options, given the field and the available people.

But, you need an advocate in setting it up and preferably at the defense as well.

Don't pick professor X because you don't need that stress. At my university you weren't even supposed to pick someone you've collaborated with for conflict of interest reasons. If anyone asks tell them you wanted the significance of your work to be evaluated by the broader audience and not by someone that might be biased in favour of your research. I think that's a very defendable position.