Detect if X11 is available (python)

Check the return code of xset -q:

def X_is_running():
    from subprocess import Popen, PIPE
    p = Popen(["xset", "-q"], stdout=PIPE, stderr=PIPE)
    p.communicate()
    return p.returncode == 0

As for the second part of your question, I suggest the following main.py structure:

import common_lib

def gui_main():
    ...

def cli_main():
    ...

def X_is_running():
    ...

if __name__ == '__main__':
    if X_is_running():
        gui_main()
    else:
        cli_main()

I'd check to see if DISPLAY is set ( this is what C API X11 applications do after all ).

import os

if os.environ.get('DISPLAY'):
    print("X11 is available")

You could simply launch the gui part, and catch the exception it raises when X (or any other platform dependent graphics system is not available.

Make sure you really have an interactive terminal before running the text based part. Your process might have been started without a visible terminal, as is common in graphical user environments like KDE, gnome or windows.