Why are hex editors called binary editors?
A binary editor edits a file as a binary file.
Binary file - Wikipedia
A binary file is a computer file that is not a text file. The term "binary file" is often used as a term meaning "non-text file" [and is] usually thought of as being a sequence of bytes. ... Binary files typically contain bytes that are intended to be interpreted as something other than text characters.
A hex editor is a type of binary editor in which the data is represented in hexadecimal.
Hex editor - Wikipedia
A hex editor (or binary file editor or byte editor) is a type of computer program that allows for manipulation of the fundamental binary data that constitutes a computer file. The name 'hex' comes from 'hexadecimal': a standard representation for numbers that has 16 as its base.
Terminology is hard. Different people have all sorts of different names for things.
In this instance, it appears that the hex in "hex editor" refers to the conventional human-readable representation of each byte's value, whereas the binary in "binary editor" refers to the notion that you're indeed editing the file at the byte layer (and computers store bytes in binary), without consideration for higher-level text encoding and the like. Recall that files not readily representable in higher-level text form are called "binary files" or "binaries", for the same reason.
Neither is technically incorrect; they just come at the naming problem from different angles. On a personal note, though, I would tend to agree that "binary editor" is confusing on balance.
Binary editor display (in binary)
00000000 | 11111011 11111011 11111011 11111011 11110101 11111011 11111011 11111011
00001000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
00010000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
00011000 | 11110101 11111011 11111011 11111011 11111011 11111011 11111011 11111011
00100000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
00101000 | 11111011 11111011 11101111 11111011 11111011 11111011 11111011 11111011
00110000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
00111000 | 11111011 11111011 11111011 11101111 11111011 11111011 11111011 11111011
01000000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
01001000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
01010000 | 11101111 11111011 11111011 11111011 11111011 11111011 11111011 11111011
01011000 | 11101111 11111011 11111011 11111011 11111011 11111011 11111011 11111011
01100000 | 11111011 11111011 11111011 11111011 11111011 11101111 11111011 11111011
01101000 | 11111011 11111011 11111011 11111011 11111011 11111011 11111011 11111011
Binary editor display (in hexadecimal)
00 | fb fb fb fb f5 fb fb fb fb fb fb fb fb fb fb fb
10 | fb fb fb fb fb fb fb fb f5 fb fb fb fb fb fb fb
20 | fb fb fb fb fb fb fb fb fb fb ef fb fb fb fb fb
30 | fb fb fb fb fb fb fb fb fb fb fb ef fb fb fb fb
40 | fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
50 | ef fb fb fb fb fb fb fb ef fb fb fb fb fb fb fb
60 | fb fb fb fb fb ef fb fb fb fb fb fb fb fb fb fb
It is true that a binary editor should let you handle bytes in binary, but as you can see the base 2 is too small to give compact numerals.
Binary as raw, not base two
Generally speaking, then "binary" in binary editor doesn't refer to the base two, it actually is the antonym of "text editor".
The difference being that text editors are designed to show text, so non printing characters, new lines, control chars, invalid utf-8 code units and so on are not required to be handled safely.
The mere act of opening a file and immediately saving it can change it (due to the editor internal conversions, failures).
Binary editors on the other way, don't try to give the data any semantic and let the user safely handle it as a stream of bytes/bits/words.
Hexadecimal editors
The perfect binary editor would known every possible binary format and let you edit it, but since everyone can make its own binary format and since they change
very often, it is futile to try to support all formats.
The best thing an editor could do is showing the bytes themselves, and due to the, already discussed, properties of base 16, hexadecimal numerals are very handy!
Also an experienced user can mentally translate bin to hex as they read it.
Setting/getting a bit out of a byte written in hex takes really absolutely no more effort than a byte written in bin.
To be honest I found hex easier than binary to read.