Terminal only as "Desktop"

One way or another, you would need X running. But you can get something like what you're asking with a tiling window manager. One of the earlier ones was "ion" (not as popular now).

Further reading (no specific recommendations, of course: that would introduce opinion):

• Comparison of tiling window managers (Arch wiki)
• Why You Should Try a Tiling Window Manager
• Exploring Tiling Window Managers
• 5 Great Tiling Window Managers for Linux

Technically, all you need to run GUI programs is the X server. You can run just a terminal emulator and run all programs from that. However life without a window manager is not comfortable at all: there's no interface to switch between, raise, resize, move, hide, close, and otherwise manipulate windows.

So what you need is a window manager, probably without a desktop environment. There are a lot of window managers out there — Wikipedia has a comparison table, Debian includes more (57 packages provide x-window-manager), but more exist (and even that list is incomplete, but most of the ones not on that list are probably minimalist programs used mostly by their author).

A large majority of less-popular window managers tend to be on the minimalist side, but not all are. Pretty much any window manager apart from the ones specifically designed for a desktop environment can be used comfortably without any kind of screen “distraction”.

If you want to have a lot of control over how your environment works, three choices stand out: awesome, which is programmed in Lua; sawfish, which is programmed in Lisp; xmonad, which is programmed in Haskell. Awesome and xmonad are tiling, sawfish is stacking (and awesome supports stacking to some extent). If you're in the habit of having full-screen windows most of the time then you'll probably like a tiling window manager. If you never have enough screen real estate to fit all the windows you want to see together then stacking is really helpful.

If you prefer something minimalist with little configuration, go through the list and test a few until you find one whose author's idiosyncrasies match yours.

No matter what window manager you choose, you can run a full-screen terminal upon login and run all your programs from there. Just about any window manager provides a way to run programs but you don't have to use that if you don't want to.