Why do here-documents attempt shell substitution even on a commented line?

This is more general than bash. In POSIX shell, your EOF is referred to as a word, in the discussion of here-documents:

If no characters in word are quoted, all lines of the here-document shall be expanded for parameter expansion, command substitution, and arithmetic expansion. In this case, the <backslash> in the input behaves as the <backslash> inside double-quotes (see Double-Quotes). However, the double-quote character ( '"' ) shall not be treated specially within a here-document, except when the double-quote appears within "$()", "``", or "${}".

Quoting is done using single-, double-quotes or the backslash character. POSIX mentions the here-documents in the discussion of quoting:

The various quoting mechanisms are the escape character, single-quotes, and double-quotes. The here-document represents another form of quoting; see Here-Document.

The key to understanding the lack of treatment of # characters is the definition for here-documents:

allow redirection of lines contained in a shell input file

That is, no meaning (other than possible parameter expansion, etc) is given to the data by the shell, because the data is redirected to another program: cat, which is not a shell interpreter. If you redirected to a shell program, the result would be whatever the shell could do with the data.

Within a here document there are no comment lines.

man bash:

No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \ is ignored, and \ must be used to quote the characters \, $, and `.

So you need:

cat <<"EOF"