Is X11 dangerous?
Any application launched from under the current user has access to the keyboard, mouse, display (e.g. taking a screenshot), and this is not good.
All the X11 clients on a desktop can access each other in depth, including getting the content of any window, changing it, closing any window, faking key and mouse events to any other client, grabbing any input device, etc.
The X11 protocol design is based on the idea that the clients are all TRUSTED and will collaborate, not step on each other's toes (the latter completely broken by modern apps like Firefox, Chrome or Java).
BUT, if we install programs from the official repository (for example, for Debian), which are unlikely to contain keyloggers, etc., then the danger problem is clearly exaggerated. Am I wrong?
Programs have bugs, which may be exploited. The X11 server and libraries may not be up-to-date. For instance, any X11 client can crash the X server in the current version of Debian (Buster 10) via innocuous Xkb requests. (That was fixed in the upstream sources, but didn't make it yet in Debian). If it's able to crash it, then there's some probability that it's also able to execute code with the privileges of the X11 server (access to hardware, etc).
For the problems with the lax authentication in Xwayland (and the regular Xorg Xserver in Debian), see the notes of the end of this answer.
Yes, you can open applications on separate servers (for example, Xephyr), but this is inconvenient, since there is no shared clipboard. Creating a clipboard based on tmp files is also inconvenient.
Notice that unless you take extra steps, Xephyr allows any local user to connect to it by default. See this for a discussion about it.
Creating a shared clipboard between multiple X11 servers is an interesting problem, which deserves its own Q&A, rather than mixed with this.
Applications running on the same machine with the same user account can use the ptrace system call to modify each other's process memory, so X11 is not the most convenient attack surface here.
For applications you don't fully trust, you need to first run them with a different user ID (like Android does with applications from different vendors), and you can use the XSECURITY extension to generate an "untrusted" access token for the X server, with which the application's access to X11 is restricted:
- no access to input events not directed at the own window
- no access to the XTEST extension
- no transparent windows
Input events in X11 have a Synthetic field that tells whether the input event was generated from an input device or sent from another program, and the receiving program decides what to do with synthetic events, for example xterm just ignores them. The XTEST extension allows generating non-Synthetic events from software for testing purposes, which is why untrusted clients are not allowed to use that extension.