Apple - Restart OS X GUI

You can also try:

killall -KILL Dock

The WindowServer is responsible for drawing all Windows and Windows Content for all Programs (incl. Finder), so it would not be possible to kill the Process without killing all Applications.

You can read about the Window Server in the Apple Technical Note TN2083 for OSX 10.5.

Even though it is announced in that Document that Apple plans to change the Behaviour in future Systemversions, it's in your list of required OS's.

There is no way to restart window server without closing all apps; logging out & then back in is the best option