Apple - What is the X Server equivalent on OSX?
What I was looking for turns out to be the display server called The Quartz Compositor that along with Quartz 2D forms the Core Graphics framework of the OSX.
X Quartz seems to be the interface that routes requests from applications that use the X11 framework to the Quartz Compositor.