How to unload automatically loaded packages?

If you want to revert the entire system to some state, then CleanSlate` may be the best option. If you want to unload a few specific packages though, you can use my package PackageManipulations`, available here. It has a function PackageRemove, which does exactly that. It has an accompanying notebook with explanations. Some additional notes on it are in this and this answers. If you want to just clear all the package and sub/package symbols but not Remove them, you can use functions PackageClear and PackageClearComplete from the same package (a disclaimer : the package may contain bugs, although I used it successfully many times)

Note that removing all symbols in the package (and subsequently removing its context) - which is what PackageRemove does and what you seem to ask for - may be unsafe if other packages use some of those symbols, since those symbols in the definitions of symbols in dependent packages would turn to Removed[symbs] and won't be usable any more, in a sutble way. To check that there are no such dependencies, you can use another package I wrote, PackageSymbolsDependencies`, available from the same place (it also has a notebook with explanations and examples).

The cleanest possible kernel you can get is starting a kernel only (no front end) with the -noinit option.

math -noinit

The front end will evaluate a lot of code when it connects to the kernel. -noinit prevents the kernel from loading initialization files (init.m). I don't believe you can get to a purer state than this unless you start surgically removing components that are practically built in (Remove, Clear, etc. applied to pre-defined symbols).