Restart GNOME without restarting Xorg?

You should be able to restart Gnome with

 gnome-shell --replace

From the manual:

DESCRIPTION
   GNOME Shell provides core user interface functions for the GNOME 3 desktop,
   like switching to windows and launching  applications.  GNOME  Shell  takes
   advantage  of  the  capabilities of modern graphics hardware and introduces
   innovative user interface concepts to provide  a  visually  attractive  and
   easy to use experience.

OPTIONS
   --replace
          Replace the running window manager

You can press Alt-F2 (IIRC) to get a window where you can type a command. In addition to regular commands, GNOME Shell supports a couple of custom commands to control the behavior of the desktop environment. In this case, you want restart, which restarts the Shell, not the computer.

If it's crashed too much for that to work, use terdon's answer.

TL;DR, if you cannot do it from the GUI just login as the same user (text console, ssh, etc) and type:

eval "$(while read -d $'\0' line; do printf 'export %q\n' "$line"; done </proc/$(pgrep -fu$USER gnome-session-binary)/environ)"
gnome-shell --replace >.xsession-errors 2>&1 &

I do it often from a text console (CTRL+ALT+Fn where n is a free console, usually starting from F3 as F1 is the login screen and F2 is my session).

It obviously won't work with Wayland since gnome-shell restarts are just not supported on it.

---

Detailed answer:

The trick is to replicate the environment required by gnome-shell - I haven't verified exactly what is needed but at the very least you need DISPLAY and XAUTHORITY so gnome-shell can connect to the X server, and then a few additional ones are very likely required by gnome-shell and its extensions.

The process tree should look a bit like this:

      |            |-gdm-session-wor(6444)-+-gdm-x-session(6464)-+-Xorg(6466)-+-{InputThread}(6472)
      |            |                       |                     |            `-{radeon_cs:0}(6467)
      |            |                       |                     |-gnome-session-b(6475)-+-evolution-alarm(6712)-+-{dconf worker}(6789)
[...]
      |            |                       |                     `-{gmain}(6465)
      |            |                       |-{gdbus}(6446)
      |            |                       `-{gmain}(6445)

In this example gnome-session-binary (pid 6475) is the parent of all gnome session processes, so this is what I use to get the environment. I use pgrep to return the pid of that process for the current user only (there should always be only one).

Then I read that task's environ file from /proc/<PID>/ (null-separated list of environment variables), and prepend all lines with export. Those commands are then fed back to my shell through eval.

At this point I can run gnome-shell --replace like if I was doing it from the GUI session. If you plan on keeping that shell open you will likely want to redirect STDOUT and STDERR somewhere too; you can send it to /dev/null but I like to keep it around just in case (.xsession-errors is the legacy file X used to throw all GUI output to; on modern systems I believe systemd or journald capture all output and logs it instead).