Lightweight X11 alternative available?

The XFree86 implementation of the X server includes TinyX, which is part of many small Linux distributions e.g. Damn Small Linux or embedded Linux distributions.

TinyX perfectly fits your requirements.

The only server implementations talking the X11 protocol I know of are XFree86 and X.Org. Note that X.Org is the server implementation shipped by most Linux distributions, due to licensing issues with XFree86. I don't see why those shouldn't run on your machine given those specs, provided that appropriate graphics drivers are available. Judging by the tags you're using Gentoo, so you should be able to just install X.Org by running emerge xorg-x11 and waiting for it to finish compiling (which might take a while on an old machine like this).

You probably won't be able to run modern desktop environments like Gnome or KDE though, especially given the memory limitations. I would give Xfce a try, or perhaps LXDE.

First, the big caveat: I think X with a lightweight desktop environment is really going to be your best bet for desktop hardware, because a) it includes wide hardware support, including 2D and 3D acceleration on a lot of old graphics cards, b) it's not really that awfully heavyweight, and c) all X programs will just work.

But there are alternatives. These generally work by running directly on the Linux framebuffer console, possibly via directfb. Some options here would be:

• Android-x86: a port of Google's phone/embedded OS to PC hardware. Linux kernel, but not necessarily a Unix-like userspace.
• Qt QWS: embedded version of the popular toolkit (apparently KDE is even partly ported)
• GTK-DFB a similar thing for GTK (now defunct)
• SDL forget all those "toolkits", with their "widgets" and "sophisticated support libraries" and "convenience"! Write your graphics as directly as possible, since SDL has direct framebuffer support

But, depending on your hardware, all of that trouble might not really get you anything, because it won't necessarily be faster. And you'll have to find ports of anything you want to run, or port it yourself.