How to resolve 'No protocol specified' for su user
The problem is not occurring because of the UID of the user. 500 is just fine as a UID, and that UID doesn't make it a 'non-login' user except in the eyes of the default settings of some few display managers.
The error message No protocol specified sounds like an application-specific error message, and an unhelpful one at that, but I am going to guess that the error is that the application is unable to contact your X11 display because it does not have permission to do so because it's running as a different user. Applications need a "magic cookie" (secret token) in order to talk to the X11 server so that other processes on the system running under other users cannot intrude on your display, create windows, and snoop your keystrokes. The other system user does not have access to this magic cookie because the permissions are set so that it is only accessible to the user who started the desktop environment (which is as it should be).
Try this, running as your original user, to copy the X11 cookie to the other account:
su - <otheruser> -c "unset XAUTHORITY; xauth add $(xauth list)"
then run your application. You may also need to unset XAUTHORITY in that shell too. That command extracts the magic cookie (xauth list) from your main user and adds it (xauth add) to where the other user can get it.
In my case the new display server protocol wayland was the problem,
just do xhost + local: then other users (e.g. root) are allowed to run programs in your session, network connections however will not be allowed.
If you want to allow clients from any host, you can use xhost + without specifying any host at all. This is however unsafe, it would be better to just specify the host(s) for which you want to grant access to your session.
Suppose you want to brute force get yourself a connection with X...
Lets assume you are already running your commands on the server (where X runs), otherwise get that to work first and then use 'ssh -X user@server' from the client afterwards ;).
There might be several ways to run the xauth commands, for example, you might be using 'sudo', but that might lose or change environment variables. The following environment variables need to be preserved: DISPLAY and XAUTHORITY. To test if that is the case you could run 'echo $XAUTHORITY' in the same way you run your commands, but make sure you aren't expanding the environment variables before you run those commands. For example, try: sudo bash -c 'echo "$XAUTHORITY"' to see what XAUTHORITY really is after you run your sudo (if it disappears you might need to add something to your sudoers file, see elsewhere).
Eventually, run the following command as the user that you want to get access with, on the server:
xauth info
This will show the 'Authority file' that will be used (/root/.Xauthority by default, for root, or something like /home/theuser/.Xauthority). If it shows the correct .Xauthority file then you don't have to worry about the XAUTHORITY environment variable actually (actually, I wouldn't know when it wouldn't, except if you want to manipulate a non-standard place of that file).
Remove that file (if it even exists):
rm /root/.Xauthority
Replace /root/.Xauthority with the correct XAUTHORITY file for your case.
Recreate it, but empty (this is needed for a lot of commands):
touch /root/.Xauthority
At this point you'll get the No protocol specified error, even if you got Invalid MIT-MAGIC-COOKIE-1 before. Find the authority file that the X server is using at the moment:
ps aux | grep Xorg
This should show something like:
root 1153 0.0 1.0 149560 44464 tty7 Ss+ dec02 0:00 /usr/lib/xorg/Xorg -nolisten tcp -auth /var/run/sddm/{ef18c483-7891-4e82-80ef-2c8f9bd79711} -background none -noreset -displayfd 17 vt7
The file name after -auth is what you need in the next command. Run this as root:
sudo xauth -f '/var/run/sddm/{ef18c483-7891-4e82-80ef-2c8f9bd79711}' list
That lists a 32 digit hexadecimal key. For example the output could be:
hostname/unix:0 MIT-MAGIC-COOKIE-1 c0eaf749aa252101a0f57d5087089db7
Use that to generate your .Xauthority file (as user who needs to login again):
xauth add $DISPLAY MIT-MAGIC-COOKIE-1 c0eaf749aa252101a0f57d5087089db7
replace 'c0eaf749aa252101a0f57d5087089db7' with what was returned by the list command for you. Now your .Xauthority should be size 51 bytes and you can connect to the X server (again).