GUI Apps without a graphical desktop

You can install X windows without installing a desktop, and run apps under bare X. It's a bit strange like that: no background, no window borders. Or you can add a minimal window manager.

Can I have Graphics on Linux without a desktop manager?

Sure. There are couple of options (which you want is not clear from question).

1. Some programs can be run in headless mode where they accept the situation where X-server is not available
2. You only need X-server running at client computer, it is not necessary at the server running the software. For example, if Firefox is installed in server you use ssh to connect to you can use ssh -X to connect and run firefox at the server => firefox UI is drawn to your computer.

Here are several headless X servers some of them are:

• virtual x server: xpra,
• nested: xnest, xserver-xephyr

Also, you can run xorg with dummy display driver.