How does Linux's display work?

Linux display

The Linux's display system, uses multiple technology, protocols, extensions, applications, servers (daemon), drivers and concepts to achieve the windowing system for instance: Xorg, Wayland, X11, OpenGL, RandR, XrandR, Screen Resolution, DPI, Display server, etc. This can be overwhelming to understand fully, but each side of it is meant for a specific purpose and they are not used all together at the same time.

X protocol

The X Window System, X11 (X version 11) is a windowing system for bitmap displays, common on Unix-like operating systems, X provides the basic framework for a GUI environment: drawing and moving windows on the display device and interacting with a mouse and keyboard. X does not mandate the user interface, this is handled by individual programs. As such, the visual styling of X-based environments varies greatly; different programs may present radically different interfaces. X originated at the Project Athena at Massachusetts Institute of Technology (MIT) in 1984. The X protocol has been at version 11 (hence "X11") since September 1987. The X.Org Foundation leads the X project, with the current reference implementation, X.Org Server, available as free and open source software under the MIT License and similar permissive licenses.

X implementation

Most Linux distribution uses X.Org Server which is the free and open-source implementation of the display server for the X Window System (X11) stewarded by the X.Org Foundation. Xorg/X alone doesn't support multiple provided features like scaling or rendering, for that Xorg uses extensions such as XFixes, RandR (RandR is managed by xrandr it can for instance setup panning, resolution or scaling), GLX (OpenGL extension), Render or Composite which causes an entire sub-tree of the window hierarchy to be rendered to an off-screen buffer, applications can then take the contents of that buffer and do whatever they like, the off-screen buffer can be automatically merged into the parent window or merged by external programs, called compositing managers to do compositing on their own like some window managers do; E.g. Compiz, Enlightenment, KWin, Marco, Metacity, Muffin, Mutter and Xfwm. For other "non-compositing" window managers, a standalone composite manager can be used, example: Picom, Xcompmgr or Unagi. Xorg supported extensions can be listed with: xdpyinfo -display :0 -queryExtensions | awk '/^number of extensions:/,/^default screen number/'.

On the other hand Wayland is intended as a simpler replacement for Xorg/X11, easier to develop and maintain but as of 2020 desktop's support for Wayland is not yet fully ready other than Gnome (e.g. KDE Kwin and Wayland support); on the distributions side, Fedora does use Wayland by default. Note that Wayland and Xorg can work simultaneously, this can be the case depending on the used configuration. XWayland is a series of patches over the X.Org server codebase that implement an X server running upon the Wayland protocol. The patches are developed and maintained by the Wayland developers for compatibility with X11 applications during the transition to Wayland, and was mainlined in version 1.16 of the X.Org Server in 2014. When a user runs an X application from within Weston, it calls upon XWayland to service the request.

The whole scope

           

A display server or window server is a program (like Xorg or Wayland) whose primary task is to coordinate the input and output of its clients to and from the rest of the operating system, the hardware, and each other. The display server communicates with its clients over the display server protocol, a communications protocol, which can be network-transparent or simply network-capable. For instance X11 and Wayland are display server communications protocols.

As shown on the diagram a window manager is an other important element of the desktop environment that is a system software that controls the placement and appearance of windows within a windowing system in a graphical user interface. Most window managers are designed to help provide a desktop environment. They work in conjunction with the underlying graphical system that provides required functionality support for graphics hardware, pointing devices, and a keyboard, and are often written and created using a widget toolkit. KDE uses KWin as a window manager (it has a limited support for Wayland as of 2020), similarly Gnome 2 uses Metacity and Gnome 3 uses Mutter as a window manager.

An other important aspect of a windows manager is the compositor or compositing window manager, which is a window manager that provides applications with an off-screen buffer for each window. The window manager composites the window buffers into an image representing the screen and writes the result into the display memory. Compositing window managers may perform additional processing on buffered windows, applying 2D and 3D animated effects such as blending, fading, scaling, rotation, duplication, bending and contortion, shuffling, blurring, redirecting applications, and translating windows into one of a number of displays and virtual desktops. Computer graphics technology allows for visual effects to be rendered in real time such as drop shadows, live previews, and complex animation. Since the screen is double-buffered, it does not flicker during updates. The most commonly used compositing window managers include: Linux, BSD, Hurd and OpenSolaris-Compiz, KWin, Xfwm, Enlightenment and Mutter. each one have its own implementation, for instance KDE's KWin's compositor have many features/settings like animation speed, tearing prevention (vsync), window thumbnails, scaling method and can use OpenGLv2/OpenGLv3 or XRender as a rendering backend along with Xorg. (XRender/Render not to confuse with XRandR/RandR).

OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. The API is typically used to interact with a graphics processing unit (GPU), to achieve hardware-accelerated rendering. OpenGL is a rendering library that can be used with Xorg, Wayland or any application that implements it. OpenGL installation can be checked with glxinfo | grep OpenGL.

The display resolution or display modes of a computer monitor or display device is the number of distinct pixels in each dimension that can be displayed. It is usually quoted as width × height, with the units in pixels: for example, 1024 × 768 means the width is 1024 pixels and the height is 768 pixels. xrandr can be used to add or render/simulate a new display resolution.

The DPI stand for dots per inch and is a measure of spatial printing/display, in particular the number of individual dots that can be placed in a line within the span of 1 inch (2.54 cm). Computer's screens do not have dots, but do have pixels, the closely related concept is pixels per inch or PPI and thus DPI is implemented with the PPI concept. The default 96 DPI mesure mean 96x96 vertically and horizontally. Additionally Is X DPI (dot per inch) setting just meant for text scaling? QA is very informative.

Notes

Some KDE's gui tools: systemsettings5 > display, kcmshell5 xserver and kinfocenter.

References

Links and sources: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12.

The question is quite broad, and a lot more could be written on the subject than this answer covers. I have tried to provide a historical perspective on the evolution of Linux graphics. Graphics, windowing systems, and graphical user interfaces (GUIs) on Linux have gone through a lot of changes since the early 1990s, when the X Window System (X11) was ported to Linux.

The X Window System

The X Window System was developed at MIT in the 1980s. The name X11 refers to protocol version 11 of the X protocol, but X10 was also used outside of MIT before being replaced by version 11 in 1987.

The X Window System was designed to work on graphics systems that were the state of the art in the 1980s. A typical workstation had a single frame buffer connected to a simple CRT controller which displayed the contents of the frame buffer on a display monitor. Computing before the PC and workstation era was done via serial line ("dumb") terminals connected to central computers situated in computer machine rooms. This historical context influenced the design of X11: graphical applications could be run on remote computers with the user interacting with the program using terminals with graphics capabilities. The "terminal" could be a workstation or a dedicated X terminal.

X11 was designed as a server-client system. The X server was the only part communicating directly with the graphics hardware. The X clients are application programs talking to the server using the X protocol, either using a local Unix Domain socket or a TCP/IP connection. The X protocol is used by the client to both send requests to the server and receive event messages from the server.

Requests include messages for:

• window creation
• mapping/unmapping a window: making the window visible/invisible
• drawing on the window: draw pixels, lines, arcs, ovals, pixmaps, etc.
• displaying text using a specified font, size and style
• moving and resizing windows, changing the stacking order of windows, etc.

Clients receive messages (not an exhaustive list):

• replies to requests
• keypress and mouse click events
• expose events (an area of a window needs to be redrawn)
• focus gain/loss events

To enable the user to handle the windows on screen, for example, move, resize, close, raise and lower a window, a particular application called the window manager is provided. The window manager can also display window decorations like borders, title bars, and global menus.

You could say the X11 server is quite "high level", as it handles (or at least traditionally handled) all kinds of resources: windows, fonts, pixmaps, colormaps, graphic contexts (things like foreground/background color, line width, etc). In addition to this, the server takes care of things like window parent-child relationships and the stacking order of windows.

The X protocol is designed to be extensible. The X server can be taught to do new tricks, and new opecodes are added to the protocol to have the server perform those tricks. For example, the XRender extension introduces a way to handle transparency ("alpha blending"). This extension was introduced mainly to support anti-aliased fonts, but has also been used for desktop effects like drop shadows on windows. The RandR ("Resize and Rotate") extension makes it possible to resize, rotate and reflect the root window on the screen. This enables you to project the screen using a projector that is upside down, or to use a tilted monitor.

The GLX extension (OpenGL Extension to the X Window System) makes it possible to use OpenGL in a window provided by the X server. The calls to OpenGL are embedded in X protocol requests.

At some point in the evolution of X11, font handling was moved to be handled by the client. The reasons behind this change is discussed in New Evolutions in the X Window System.

Direct Rendering

In the early 2000s, display hardware had come a long way from the simple black-and-white bitmapped displays that existed when development of X started in the 1980s. The X11 relative overhead of the inter-process communication (IPC) model had grown too big, even when using a local socket. The solution to this was to abandon the principle that the X server is the only part that talks directly to the hardware, and let the clients talk to the graphics card directly. The Direct Rendering Infrastructure (DRI) was born.

DRI allows an X client app to bypass the X server and render directly on the graphics adapter. Because several direct rendering applications in addition to the traditional X server can be active at the same time, a kernel component called the Direct Rendering Manager was introduced to arbitrate access to hardware. There are three versions of the DRI architecture, the original DRI (obsolete), DRI2, and DRI3.

Compositing Window Managers

The next innovation to enter the Linux graphics scene was the compositing window manager. Traditionally, each X client application was responsible for repainting its windows (partially, or the whole window) on demand. The X server sent the application an Expose event when a repaint was needed as the result of the window being mapped on the screen, or if it is no longer obscured by some other window. When an overlapping window is removed, the window beneath it is exposed. Failing to repaint this area lead to the old contents still being displayed. https://en.wikipedia.org/wiki/Visual_artifact

A compositing window manager changes this. Applications render to their own off-screen buffers, each of which is kind of a separate screen with exclusive access by the application owning the buffer. It is the task of the compositing window manager to display these buffers in windows on a real screen, clipping any windows that are obscured by other windows or partially off-screen. The window manager displays a "composition" of the windows.

A compositing manager can typically also display animated effects, like scaling, warping, fading, rotating, blurring the windows. For example, moving a window can make it wobble, or virtual desktops can be displayed on the side of a rotating cube.

Kernel Mode Setting

The X server traditionally also took care of setting the modes of the graphics adapter, like resolution and refresh rates. The mode setting has since been moved to a Linux kernel component called Kernel Mode Setting (KMS). This solved a lot of problems with switching between Linux's virtual consoles.

Evdev

The X server also had knowledge of the input devices and, for example, the type of mouse had to be specified in the X configuration. The X server has been relieved of this task with the introduction of the evdev subsystem of the Linux kernel, which provides a generic input event interface.

Wayland

With all these developments, a lot of the tasks performed by the X server have moved outside of the X server. Using direct rendering, clients don't use the X protocol anymore. Thanks to KMS, the X server does not need to muck with the low level programming of graphics adapters. With evdev, input device handling was simplified in the X server. When using a compositing window manager rearranging and warping windows, the X server has no idea what's going on on the screen anymore. "The window manager is the new X server".

Wayland came about as a result of the realization that the X server process had little left to do, and by cutting out the middle-man (the X server), a much simpler desktop graphics system could be achieved. Backward compatibility is provided via Xwayland, a modified Xorg server that displays top level X windows using Wayland surfaces.

Strictly speaking, Wayland is just a protocol that defines how clients communicate with the display server. The Wayland protocol is quite unlike the X protocol: the Wayland protocol does not define messages to draw graphics or text, nor does it handle fonts.

In the Wayland architecture, the window manager and the display server are merged into one software component, the compositing window manager. Clients can request, via a software library using the Wayland protocol, a surface to draw on. A "surface is an object representing a rectangular area on the screen, defined by a location, size and pixel content".

Clients render into off-screen buffers, which are then attached to a surface, producing output on the screen. The client can use various APIs to do the rendering: OpenGL, OpenGL ES, etc. ("What is the drawing API? Whatever you want it to be") Double buffering is used: a client updates its image using a second buffer, and when that buffer contains a coherent image, it is switched to be displayed at the next display monitor vertical blanking interval. Wayland's motto is: "Every frame is perfect", i.e. windows do not tear, flicker or flash.

Input handling in Wayland goes through the compositor, which is the only component knowing which window is under the mouse cursor (remember, the compositor may also have warped the windows). The compositor transforms the screen coordinates to window-local coordinates of the appropriate window and sends the event to the client.

If you are interested in the story that led to the creation of Wayland, I recommend watching Daniel Stone's hilarious presentation The real story behind Wayland and X.