How can I run UI tests with FlaUI on a remote machine whilst not RDP'd into it?

If you simulate a mouse click, there has to be an active desktop session (https://github.com/Roemer/FlaUI/wiki/FAQ#how-can-i-run-flaui-tests-on-a-build-serveragent).

You have two options: test without mouse clicks (use UIA patterns) or ensure an active desktop session for the build agent. As stated in the FAQ, make sure the session is not closed after disconnecting RDP by running tscon 1 /dest:console