A way for DNS record to say "this domain has no mail server"?

Due to the fallback of directly contacting a host via its address records, a single "null MX" record of "MX 0 ." is the apparent preferred way to indicate that the host does not accept e-mail. This is similar to a "null SRV" record ("SRV 0 0 0 .") which specifically marks a service as not available (per the SRV-RR RFC 2782).

This has been standardized by RFC 7505 (as of December 2017 it is a proposed standard).

"MX 0 localhost." (or equivalent label pointing at ::1 and 127.0.0.1) is also acceptable but more appropriate for a host that must send mail to itself (e.g. cron job output) which does not accept external mail. Such hosts may have an operational mail server which is firewalled off from the Internet but other services are accessible.

Having no MX record and blocking the SMTP port does not stop people from wasting one's incoming bandwidth trying to contact a non-existent server. The single MX record methods above do prevent such traffic because address-type records are never tried when at least one MX record is present. This will probably not stop some spammers from trying to contact a host directly via its address records. However, as it does stop legitimate traffic from trying, you'll be able to identify spam sources with 100% certainty.

Using private addresses should not be used because one cannot tell where they'll end up. Using other reserved addresses (e.g. documentation address of 192.0.2.0/24) is also inappropriate except where trying to identify and trap spammers within one's own network when they try to connect.

I don't know what the "standard" way is, but here's one I've run across: set an MX record to a loopback address.

I suppose any private IP address (or otherwise "invalid" IP - 0.0.0.0) would do the trick. I personally think it's kind of a lousy thing to do, but it would do what you want. You could couple it with a hostname like thisdomaindoesntacceptemail.sostopsendingit as a service to the mail admin who'll end up with the ticket for "email being down" because your domain won't accept email. :)

However, why not just remove the MX record, and set your firewall rules on the A record to block SMTP and TLS (and any other mail ports)?

That would get the point across, and any admin who does a lookup will see no MX record, and refused connections on the fallback A record will remove any doubt about the intent of your configuration, should anyone even look more closely after seeing no MX record.

A simple TXT record will do this for you, set the SPF records to have a null value with a hard fail:

@ IN TXT "v=spf1 -all"
* IN TXT "v=spf1 -all"

That's how I ensure a domain can't be phished that I use to internal or non-mail services.