Why removing the xfce4 package doesn't remove the XFCE desktop environment?

xfce4 itself is a meta-package that will install a default configured xfce desktop environment.

In most cases the base files that comes with xfce4 are: xfconf, xfce4-utils, xfwm4, xfce4-session, thunar, xfdesktop4, exo-utils

So you can do

sudo apt-get purge xfconf xfce4-utils xfwm4 xfce4-session thunar xfdesktop4 exo-utils xfce4-panel xfce4-terminal

Then most of the package that were associated with these package become autoremovable so you can run

sudo apt-get autoremove

or, almost all xfce4 package depend upon libxfce4util-common. Just purge that one and you remove everything related to xfce

Note that apt-cache is not showing you installed packages, but rather those which are known about.

To test what you have installed, you might like to use:

dpkg --get-selections | grep xfce

Be sure to look at the second output column to see whether it's installed or deinstalled.

After purging all xfce packages, the above command returns no output on my machine.

Aptitude can do this by using package search ~n and using -P to prompt when adding or removing packages.

sudo aptitude -P remove ~nxfce4

Substitute remove with purge if you would like to remove the system configuration files:

sudo aptitude -P purge ~nxfce4

To remove user config files (use -r with care - recursive delete!!):

sudo rm -r ~/.config/xfce4