Why does Ctrl-D (EOF) exit the shell?

The ^D character (also known as \04 or 0x4, END OF TRANSMISSION in Unicode) is the default value for the eof special control character parameter of the terminal or pseudo-terminal driver in the kernel (more precisely of the tty line discipline attached to the serial or pseudo-tty device). That's the c_cc[VEOF] of the termios structure passed to the TCSETS/TCGETS ioctl one issues to the terminal device to affect the driver behaviour.

The typical command that sends those ioctls is the stty command.

To retrieve all the parameters:

$ stty -a
speed 38400 baud; rows 58; columns 191; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 1; time = 0;
-parenb -parodd cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

That eof parameter is only relevant when the terminal device is in icanon mode.

In that mode, the terminal driver (not the terminal emulator) implements a very simple line editor, where you can type Backspace to erase a character, Ctrl-U to erase the whole line... When an application reads from the terminal device, it sees nothing until you press Return at which point the read() returns the full line including the last LF character (by default, the terminal driver also translates the CR sent by your terminal upon Return to LF).

Now, if you want to send what you typed so far without pressing Enter, that's where you can enter the eof character. Upon receiving that character from the terminal emulator, the terminal driver submits the current content of the line, so that the application doing the read on it will receive it as is (and it won't include a trailing LF character).

Now, if the current line was empty, and provided the application will have fully read the previously entered lines, the read will return 0 character.

That signifies end of file to the application (when you read from a file, you read until there's nothing more to be read). That's why it's called the eof character, because sending it causes the application to see that no more input is available.

Now, modern shells, at their prompt do not set the terminal in icanon mode because they implement their own line editor which is much more advanced than the terminal driver built-in one. However, in their own line editor, to avoid confusing the users, they give the ^D character (or whatever the terminal's eof setting is with some) the same meaning (to signify eof).

CTRL_D is just a signal saying that this is the end of a text stream. You do not end a file with it, you end your input stream by typing it. Also CTRL_D does not stand for any character or byte as you can find out with the tool hexdump:

# cat >test.txt
asdf# hexdump -C test.txt 
00000000  61 73 64 66                                       |asdf|
00000004
# ll test.txt 
-rw-r--r-- 1 root root 4 Jan 21 11:55 test.txt