How to fix non-responsive Ubuntu system?

If you want a way to reboot, without saving open documents, but without hitting the reset button, then there are ways that are less likely to cause data loss. First, try Ctrl+Alt+F1. That should bring you to a virtual console, as ixtmixilix said. Once you're in a virtual console, Ctrl+Alt+Delete will shut down and reboot the machine.

If that technique doesn't work, there's always Alt+SysRq+REISUB.

As for fixing the problem without rebooting, without more information about what is going on, it would be difficult to give a good answer. If you could describe the circumstances under which this occurs (the best way to do that is to edit your question to add the information), then that may help people to give good answers. The other thing to consider is that, if your computer is becoming unresponsive--especially if it takes more than a a few seconds for Ctrl+Alt+F1 to bring up a virtual console--then you almost certainly have a bug, and by reporting it you can both help the community and maybe get an answer.

GUI Glitches Causing Unresponsive WM or X11/Wayland

This might be happening due to an interaction between an application and a window manager--or the X11 server or Wayland. A sign that this is the nature of the problem is if an application stops responding and prevents you from entering input with the keyboard or mouse to other application windows. (No application should be able to do this; some GUI component must have a bug in it for this to occur.) If that's what's happening, then you can kill the offending process in a virtual console (as ixtmixilix alluded to):

1. Press Ctrl+Alt+F1.

2. Log in. You won't see anything as you enter your password. That's normal.

3. Use a utility like ps to figure out the offending program's process name. Sometimes this is easy in Ubuntu, and other times it isn't. For example, the name of an Archive Manager process is file-roller. If you have trouble figuring it out, you can usually find the information online without too much trouble (or if you can't, you can post a question about it).

   You can pipe ps's output to grep to narrow things down. Suppose it was Archive Manager that was causing the problem. Then you could run:

    ps x | grep file-roller
   

   You'll see an entry for your own grep command, plus an entry for file-roller.

4. Attempt to kill the offending process with SIGTERM. This gives it the chance to do last-minute cleanup like flushing file buffers, signaling to remote servers that it is about to disconnect (for protocols that do that), and releasing other sorts of resources. To do this, use the kill command:

   kill PID

   where PID is the process ID number of the process you want to kill, obtained from running ps in step 3.

5. SIGTERM is a way to firmly ask a process to quit. The process can ignore that signal, and will do so when malfunctioning under certain circumstances. So you should check to see that it worked. If it didn't, kill it with SIGKILL, which it cannot ignore, and which always works except in the rare case where the process is in uninterruptible sleep (or if it is not really running, but is rather a zombie process).

   You can both check to see if the process is still running, and kill it with SIGKILL if it is, with just one command:

   kill -KILL PID

   If you get a message like kill: (PID) - No such process, you know killing it with SIGTERM worked. If you get no output, you know SIGTERM didn't work. In that case, SIGKILL probably did, but it's worth checking by running it again. (Press the up arrow key to bring up previous commands, for ease of typing.)

6. In rare instances for your own processes, or always with processes belonging to root or another user besides yourself, you must kill the process as root. To do that, prepend sudo (including the trailing space) before the above kill commands. If the above commands don't work or you're told you don't have the necessary access to kill the process, try it as root with sudo.

(By the way, kill -KILL is the same as the widely popular kill -9. I recommend kill -KILL because SIGKILL is not guaranteed to have 9 as its signal number on all platforms. It works on x86, but that doesn't mean it will necessarily work everywhere. In this way, kill -KILL is more likely to successfully end the process than kill -9. But they're equivalent on x86, so feel free to use it there if you like.)

If you know there are no other processes with the same name as the one you want to kill, you can use killall instead of kill and the name of the process instead of the process ID number.

A Process Monopolizing CPU Resources

If a process runs at or very near the highest possible priority (or to state it more properly, at or near the lowest possible niceness), it could potentially render your graphical user interface completely, or near-completely, unresponsive. However, in this situation, you would likely not be able to switch to a virtual console and run commands (or maybe even reboot).

If a process or a combination of processes running at normal or moderately elevated priority are slowing your machine down, you should be able to kill them using the technique in the section above. But if they are graphical programs, you can likely also kill them by clicking the close button on their windows--the desktop environment will give you the option to kill them if they are not responding. If this doesn't work, of course you can (almost) always kill them with kill -KILL.

I/O Problems

Buggy I/O can cause prolonged (even perpetual) unresponsiveness. This can be due to a kernel bug and/or buggy drivers. A partial workaround is to avoid heavy and simultaneous read and/or write operations (for example, don't copy two big files at once, in two simultaneous copy processes; don't copy a big file while watching an HD video or installing an OS in a virtual machine).

This is obviously unsatisfactory and the real solution is to find the problem and report it. Unless you're running a mainline kernel from kernel.org, kernel bugs should be reported against the package linux in Ubuntu (since Ubuntu gives special kernel builds that integrate distro-specific patches, and bug reports not confirmed against a mainline kernel will be rejected at kernel.org). You should do this by running ubuntu-bug linux (or apport-cli linux) on the affected machine. See the Ubuntu bug reporting documentation first; it explains how to do this properly.

Graphics Card Problems

Some GUI lockups can be caused by graphics card problems. There are a few things you can try, to alleviate this:

1. Search the web to see if other people have experienced similar problems with the same video card (and/or make and model of machine) on Ubuntu or other GNU/Linux distributions. There may be solutions more specific than what I can offer in this answer, without more specific information than is currently in your question.

2. See if different video drivers are available for you to try. You can do this by checking in Additional Drivers; you can also search the web to see what Linux drivers are available for your video card. Most proprietary video cards are Intel, AMD/ATi, or Nvidia (click those links to see the community documentation on installing and using proprietary drivers for these cards in Ubuntu). For Intel, you're best off sticking with the FOSS drivers that are present in Ubuntu, but there's still helpful information you can use. Regardless of what card you have, this general information may help.

   If you're currently using proprietary drivers, you can try using different proprietary drivers (for example, directly from NVidia or AMD/ATi), or you can try using the free open source drivers instead.

3. Try selecting a graphical login session type that doesn't require/use graphics acceleration. To do this, log out, and on the graphical login screen click the Ubuntu logo or gear icon near your login name. A drop-down menu is shown. Change the selection from Ubuntu to Ubuntu 2D. This makes you use Unity 2D instead of Unity. (If you're using GNOME Shell, you can select GNOME Fallback / GNOME Classic instead.) If in doubt and there's a selection that says "no effects," pick that, as that's probably the safest.

   This question has some more information about different graphical interfaces you can choose between in Ubuntu.

4. In newer versions of Ubuntu, you can choose between X.org and Wayland on the login screen. Whichever you've been using, try the other. Sometimes a problem with Wayland can be fixed by using X.org, or vice versa.

5. Report a bug.

Hopefully the information above has conveyed some general information about what could be causing this kind of problem. It should also serve to illuminate what kind of information might be useful for you to add to your question (depending on the specific details of the problem), to make it possible to get an even better answer. (Or to improve this answer with additional information specific to your situation.)

Yes, the easiest way to archieve this, is enable the keys to kill Xorg server, and re-launch X session.

To enable this, open System Settings -> Keyboard Layout options (or use keyboard command in Unity interface, hit AltF2), and check this option:

Ctrl + Alt + Backspace

So when it hangs, hit CtrlAltBackspace will kill X server, and lightdm will re-launch login interface