Remove packages to transform Desktop to Server?

To remove x11 and everything that uses it, including all configuration:

apt-get purge libx11.* libqt.*

Unlike the accepted answer (remove xserver-xorg-core) this one will also remove all desktop packages like Firefox etc.

Right autoremove only removes packages that were installed automatically as part of other package installs. The things installed in the image are seeded and so all are basically considered "manual".

You'll probably get rid of most of them by getting rid of Xorg:

sudo apt-get remove xserver-xorg-core

And then do another autoremove.

There will likely be other things that are left around that you don't need.. but .. really.. why not just ask your VPS provider for a server image.. isn't it a little silly that they even offer the desktop image?