How does X11 interact with the kernel / perform login

The shell uses a TTY device (if it’s connected to one) to obtain user input and to produce output, and not much else. The fact that a shell is connected to a TTY is determined by getty (and preserved by login); most of the time the shell doesn’t care whether it’s connected to a TTY or not. Its interaction with the kernel happens via system calls.

An X11 server doesn’t know about logins (just like a shell). The login process in X11 works in two ways:

• either the user logs in on the terminal, and then starts X (typically using startx);
• or an X server is started with a “display manager” which prompts the user for a login and password (or whatever authentication information is required).

The way X11 servers obtain input and produce output is very different compared to a shell. On the input side, X knows about devices that shells don’t, starting with mice; it typically manages those directly with its own drivers. Even for keyboards, X has its own drivers which complement the kernel’s handling (so as I understand it, on Linux for example X uses the TTY driver to read raw input from the keyboard, but then interprets that using its own driver). On the output side, X drives display devices directly, with or without the kernel’s help, and without going through a TTY device.

X11 servers on many systems do use TTY devices though, to synchronise with the kernel: on systems which support virtual terminals, X needs to “reserve” the VT it’s running on, and handle VT switching. There are a few other subtleties along the way; thus on Linux, X tweaks the TTY to disable GPM (a program which allows text-mode use of mice). X can also share a VT...

On some workstations in the past, there wasn’t much explicit synchronisation with the kernel; if you didn’t run xconsole, you could end up with kernel messages displayed in “text mode” over the top of your X11 display.

The kernel has absolutely no concept of a login, or of who is logged in - all it cares about is that there are processes running under a certain user ID (who that is is also none of the kernel's business!). A login program, be it getty on a terminal or an X11 display manager, runs as root and then uses setuid() and related calls to setup an evironment for whatever UID the local resolver (NSS and libc stuff - not kernel business either) suggested to use for the user that just authenticated.