Why is DNS-over-HTTPS such a big security nightmare compared to DNS-over-TLS?

Again, it's all about the threat model!

Technologies are just technologies and can be used both for good and for evil. DNS over HTTPS (DoH) intends to solve the privacy concerns there are with unencrypted DNS, whereas DNSSEC can solve the integrity concerns without a need for encryption. Together with DNS over TLS (DoT) they are all fighting the threath of a malicious network operator that spies on your DNS traffic or forges responses.

On the other hand, both monitoring the DNS traffic and forging records can also be used for good intentions like detecting and blocking malicious traffic that is depending on DNS resolution. This is where solving these technical threats can actually decrease overall security, especially on corporate networks. The threat models of an organization are naturally different from the threat models of any individuals working for the organization.

Not so easy to detect

It's true that being unable to block DoH is rather irrelevant, if VPN connections are allowed. On corporate networks VPN connections (as well as DoH) can be either forbidden by policy (weak) or blocked by TLS inspection (efficient, but sometimes illegal or requires special privacy considerations).

Compared to DoH, DoT is easy to block, as it has a dedicated port 853 (tcp&udp) per RFC 7858.

For more detailed insights on the subject I'd recommend: Drew Hjelm: A New Needle and Haystack: Detecting DNS over HTTPS Usage (SANS Institute 2019). It also has some examples of the real security issues:

2.3. Public Threats from Encrypted DNS

Organizations need to start evaluating the risk associated with the DoH protocol because attackers have already begun using DoH to look up command-and-control (C2) servers. The best-known example of DoH as a C2 mechanism came in April 2019 with the Godlua backdoor (360 Netlab, 2019). A newer variant of the Godlua backdoor runs on Linux and Windows and uses a DoH request to grab a part of its C2 information.

Another way an attacker could use DoH in an attack is to trigger a redirected webpage as part of a spam campaign. Researchers at MyOnlineSecurity (2019) found a sample where an email attachment had a Base64 encoded string that would query Google DoH for a TXT record. The TXT record would have a JavaScript redirect to a spam webpage whose address often changed.

Numerous DoH C2 proofs-of-concept are publicly available, meaning that the threat of malicious actors using DoH is likely to increase soon.

I've heard the argument against DNS-over-HTTPS that it is supposed to be a security nightmare for network defenders because it enables encrypted DNS over port 443, compared to DNS-over-TLS which goes through port 853.

These network defenders are possibly corporate environments that rely on plaintext DNS inspection to enforce policies. Assuming that devices fallback to plaintext DNS if DoH/DoT are unavailable, the network administrators could block port 853 with little risk because it is only used by DoT. On the other hand, if they simply block port 443, then all HTTPS websites will become unavailable.

Similarly, if they see an influx of DoT traffic, it could indicate an anomaly. If some similar traffic spikes occur with DoH, then it might not be possible to directly distinguish HTTPS from DoH traffic.

As for the question from the title:

Why is DNS-over-HTTPS such a big security nightmare compared to DNS-over-TLS?

This should probably be worded as "Why is DNS-over-HTTPS seen as a security nightmare compared to DNS-over-TLS?". DoH and DoT are pretty similar on a protocol level, in both cases DNS messages are encrypted. See also my Cloudflare blog post explaining DNS encryption where I describe the technical protocol details, deployment choices, and various expectations from individuals and organizations.

Historically, the operating system has been accepting whatever DNS resolver was advertised by the local network. This is typically configured by the corporate network administrator, or the ISP. They expect to have the ability to provide services such as malware blocking, parental filtering, blocking of illegal content, and in some cases query logging.

DoH and DoT are great in protecting the privacy and integrity of DNS queries in untrusted environments such as airport Wi-Fi or even snooping/interference from the local government. However since it was emerging technology, not all existing DNS resolvers have support for it.

That put early adopters such as Mozilla in a difficult position, should they abandon the idea of improving privacy, or should they select a DNS resolver who supports DoH with a strong privacy policy? They ended up with the latter, but that meant that the default DNS resolver provided by the operating system was initially ignored. This is probably the reason for the negative pushback against DoH from ISPs and governments. If DoT was deployed in a similar way, I would have expected a similar criticism.

To conclude, I don't think that DoH is such a "security nightmare" as claimed. It is just that some organizations are concerned about losing control over DNS. Previously it was centrally controlled by the operating system, but as DoH/DoT is still pretty new, there is no real standard on configuring it so many applications have their own mechanisms to do so. This is probably the "nightmare" that some admins have, the extra complexity that they have to go through to ensure that their filtering policies are applied.

You're right that their argument makes no sense, but it's not supposed to. It's just supposed to derail DNS-over-HTTPS, which is the approach that is actually taking off because it's less likely to be blocked by existing middlebox junk. Assuming the queries are to well-known open public nameservers, it's equally easy to add new rules to block them regardless of which protocol is used, but the people who are fighting against DNS-over-HTTPS are fighting against the normalization of DNS queries being private; once that achieves critical mass as the default in browsers and client applications, blocking it will just break everything, making it practically unblockable.