How should I debug the error "Could not grab keyboard. A malicious client may be eavesdropping on your session."

Here's how to solve your mystery. The goal is more to teach users "how to fish" by using standard Ubuntu utilities to dig into the details of any process on their system.

Step #1 (for curiosity mostly): identify which program is giving you this error:

# -- You may need to search under more dirs, YMMV
#    List files (incl. binaries) which contain the warning string

$ sudo grep -ral 'malicious client may be eavesdropping' /usr /bin /lib
/usr/lib/openssh/gnome-ssh-askpass

In my env the only program which contains this warning string in its binary is gnome-ssh-askpass. I could search if there's a bug filed on this particular program, and even download its source apt-get source ssh-askpass-gnome (note that the package name is different than the program name) for further inspection.

However, I suspect the root cause is not a problem in gnome-ssh-askpass. Since gnome-ssh-askpass is asking for your passphrase, its developers simply chose to err on the side of caution when failing to grab the keyboard, assume the worst-case scenario, and make the message sound uber-paranoid. But note that typing your passphrase or password into some random web-site dialog-box by accident is probably not a good idea, so in that sense the gnome-ssh-askpass developers have made the right call.

Recently more and more web sites have started engaging in the practice of displaying a popup, fading everything else outside the popup dialog, and aggressively grabbing focus. This might be the root cause for gnome-ssh-askpass failing to grab the keyboard. If your browser being open on such site, closing the browser or navigating away from the aggressive web site may help. If this is the cause, you may be interested in a desktop setting preventing individual processes from grabbing the complete (full desktop) focus. In KDE for example, this setting can be found under (System-Settings -> Window-behavior -> Focus -> Focus stealing prevention). If you feel really paranoid, I would recommend setting it to High or Extreme. Of course, this may also prevent gnome-ssh-askpass itself from grabbing the keyboard, or more accurately: grabbing the X focus.

Step #2: Identify suspicious processes:

Knowing that in Unix, devices appear like files uder /dev, the next question is what device represents "the keyboard" in the file-system hierarchy. We can use the lsof (list open files) utility for this.

# look for processes holding devices open, filter out some common ones:
$ sudo lsof | grep /dev | grep -vE '/(null|urandom|zero)'

Notice that most of the processes holding devices open in a typical desktop env are holding /dev/pts/<N> (a pseudo tty) open. These are the "devices" of interest.

Some background on what's going on here:

In a typical Linux graphical desktop, processes don't talk to the keyboard directly. Instead the X program (Xorg) controls all the keyboard events via a device /dev/input/event<N>. X uses an event handler (evdev) which among other things, handles keyboard events. You can also verify this by looking at the X log: /var/log/Xorg.0.log where keyboard is mentioned.

The keyboard events are forwarded from the X event handler to the process that has the mouse pointer focus at any time via the process standard-input which is open on /dev/pts/<N>. Strictly speaking: a process doesn't actually "grab the keyboard", the keyboard is held by X, the process only has (or grabs) "focus" or the attention of X so X can forward keyboard events to it via an open stdin file descriptor on /dev/pts/<N>.

Step #3: which process has the Xorg focus at any particular time?

How to figure which process has the focus at any particular time? Here's a askubuntu question answering this:

find out the application under the mouse

The summary of the answer is to run a script like the following in a terminal while navigating around with the mouse:

#!/bin/bash
# Print the process tree of the window currently in focus.
# prereqs:
#   sudo apt-get install xdotool psmisc

while true; do
   pstree -spaul $(xdotool getwindowpid "$(xdotool getwindowfocus)")
   sleep 2
done

Step #4: dig deeper into process activity

Once you have a suspect process identified, the last step is to investigate this individual process. For that you may turn to the Linux /proc filesystem (man 5 proc).

Almost anything you may want to know about a process is available under /proc. In fact, programs like lsof (list open files), debuggers which examine process state, and process-listing utilities like ps or top, all rely on /proc which is populated by the kernel, for data.

Using proc you can find where the process executable program is on disk (e.g. any program outside the standard system directories, especially if it is trying to hide under a "don't pay attention to me" kind of name, may be suspect) and using a debugger or system call tracer you can examine what exactly are they doing on the system call level (even if you don't have their source code).

Steps #2 and #3 should give you all the process-IDs (PIDs) that can potentially be reading your keyboard. For each of these PIDS (let's denote each one as $pid) you may:

Map $pid to its full command line:

cat /proc/$pid/cmdline

Map $pid to its on disk executable:

ls -l /proc/$pid/exe

Map $pid to its current working directory:

ls -l /proc/$pid/cwd

Map $pid to its original environment

cat /proc/$pid/environ | tr '\000' '\012'

Trace $pid (and its children-procs) system-call activity in real-time:

strace -f -p $pid

(There's more: see man 5 proc)

If you see an unfamiliar process that reacts to every key-press by storing it into a file (via write) or sending it over the network to via sendto, you may have found a keyboard sniffer.

You can also check which processes have (tcp+udp) network endpoints open:

# See 'man netstat' for details on all options used below
$ sudo netstat -tunapee

Bottom line:

The most likely cause for the error is not malware, but multiple processes trying to get keyboard control at the same time. One of the two is gnome-ssh-askpass (the one printing the error). The other may be an open browser on a site with an aggressive focus-acquiring dialog box.

Even in the remote chance that you actually have some malware installed, the good news is that since you're on Linux, all processes are transparent for you to research and inspect. It would be very hard for malware to really hide from you, or to prevent you from easily locating it using the above techniques, killing its processes, and removing all its files.