Need small distro without a desktop or windows manager, just to run a single graphical app

Have a look at TinyCore Linux. It comes in two variants, one CLI and one including X.

The X version including a window manager and a simple desktop is about 12MiB. If you don't need a window manager, you can just start X and your application. A window manager is not required.

You could also try the aptly named Damn Small Linux:

Damn Small Linux is a very versatile 50MB mini desktop oriented Linux distribution.

Damn Small is small enough and smart enough to do the following things:

• Boot from a business card CD as a live linux distribution (LiveCD)

• Boot from a USB pen drive

• Boot from within a host operating system (that's right, it can run inside Windows)

• Run very nicely from an IDE Compact Flash drive via a method we call "frugal install"

• Transform into a Debian OS with a traditional hard drive install

• Run light enough to power a 486DX with 16MB of Ram

• Run fully in RAM with as little as 128MB (you will be amazed at how fast your computer can be!)

• Modularly grow -- DSL is highly extendable without the need to customize

I recommend Arch Linux.

Doesn't include X or any kind of graphical interface, but you can easily install if you like, it is also tiny (529 MB), it has great support and the flexibility to whatever you want.