OBS is not working with Wayland display server

OBS Studio will not work without Xorg, as it directly talks to X11 in order to perform screen recording. There is already an open bug on the issue of recording under Wayland sessions, on the bug tracker.

You may be able to run the app under Xwayland and force QT_QPA_PLATFORM=xcb for OBS STudio, but there is no guarantees it will work.