Are there formal or informal hiring bans in some academic settings based on gender and/or nationality?

You are unlikely to get examples about formal hiring bans, in particular in western societies. It would not only be frowned upon, as in this case; it would also be against university policies, if not downright illegal. The only formal hiring bans I could think of are those implied by more general rules. The most obvious case coming to mind is that about 20 countries don't allow people with Israeli passport to enter, which means they cannot study or teach there either (about half a dozen or so countries extend this to people with other passports but evidence of a travel to Israel).

Unfortunately, it is possible that the professor was speaking a truth about what some consider in their hiring decisions...where usually they would only have the common sense to not explicitly admit the true reasons for their decisions. They would start where this professor is now: claiming that their labs are full. This is human nature, and academics are no better or worse than their fellow humans dealing with their fear and prejudice in regrettable ways.

Some positions at the University of Melbourne (in a mathematics department) are currently only open to women.

The Australian National University has an astrophysics position only open to women.

There are bans rooted in funding and grants requirements. Its not only about Iranian and energy and aerospace.

ITAR, Export Control and Defense Budget or National Labs funding are limiting most non-US citizens. NATO citizens can be accepted in some cases but really outside of these two category, other scientists will face limitations at some point of their academic career.

During hiring for faculty positions, Departments are very interested to know about the funding prospects of candidates. In Engineering schools I have seen that the nationality and various Export Control and ITAR and unavailability of many federal(but defense or energy related) grants shortens the prospects on candidates and effectively removes them from the pool of candidates.

Some engineering schools among others are dependent on defense-based labs and collaboration and they seem to mention the area of research they are interested which in effect is a signal to save candidates time if they are very focused on defense funding. As of "ban" I have seen that candidates from certain sanctioned countries can not work on certain topics(energy, nuclear, nano-material and so on), also if you consider the visa processing as a part of hiring that needs to be done by university you can imagine the bans from department of state/homeland/treasury can play a role in university hiring. All these limitation on funding and visa can flow down to the level of students and can go up to the whole department level. Some professors do not want to get involved in possible complexities or limitation and you see them not accepting certain nationalities. or departments(e.g. aerospace, nuclear) do not accept students from certain countries. Last, but not least, and aside from funding, sometimes departments care about their alumni being able to work in sectors they are training for in the US, which also will not work out with many nationalities.