What is the absolute lowest level of "drawing" abstraction in GNU/Linux?

no extra assumptions except the presence of the device driver for the monitor.

That means you can use X or Wayland, because those are the graphics driver infrastructure on Linux.

Linux (the Kernel) by itself doesn't contain any graphics primitives. It provides some interfaces to talk to the GPU, allocate memory on it and configure the on-screen framebuffer. But except raw framebuffer memory access the Linux kernel does have no way to perform drawing operations. For that you need some infrastructure in userspace.

Wayland builds on top of DRI2, which in turn talks to the DRM Kernel-API. Then you require GPU dependent state tracker. Mesa has state trackers for a number of GPUs and provides OpenGL and OpenVG frontends.

The NVidia and ATI propiatary, closed source graphics drivers are designed to work with X only. So with those to make use of the GPU you must use X. That's the way it is.

Outside of that you can manipulate the on-screen framebuffer memory through /dev/fbdev, but that's mere pixel pushing, without any GPU acceleration.