Is there anything like xvfb or xnest for Windows?

You can use "Desktops" application from SysInternals package. Run UI tests on one desktop and work on another.

I found a freeware program called "RunProcess.exe" written by Frank P. Westlake that can run a process bound to the desktop of a non-interactive (e.g. off-screen) WindowStation. However, there is no source.

Therefore I've implemented my own version (simpler than Frank Westlake's) with Java and JNA. It's in the Window Licker repository in the tools module. The class is called com.objogate.wl.win32.RunOnDesktop. At some point I may port it to C and make it a normal command-line .exe, or maybe turn it into an Ant task for running tests off-screen.