How I can find out if a Linux system uses Wayland or X11?

X11 uses the DISPLAY environment variable to find the X server. Wayland uses WAYLAND_DISPLAY. Look for the Wayland variable first. Then if you don't find it or you can't connect go on to using X11.

Do not skip checking the WAYLAND_DISPLAY variable or assume Wayland is running on "wayland-0". Some people want to use nested compositors, which you would bypass. Other people may be running Wayland but want to force X11 rendering by deleting the WAYLAND_DISPLAY variable.

use the environment variable XDG_SESSION_TYPE

on x11

echo $XDG_SESSION_TYPE
x11

on wayland

$ echo $XDG_SESSION_TYPE
wayland