Who's consuming my inotify resources?

It seems that if the process creates inotify instance via inotify_init(), the resulting file that represents filedescriptor in the /proc filesystem is a symlink to (non-existing) 'anon_inode:inotify' file.

$ cd /proc/5317/fd
$ ls -l
total 0
lrwx------ 1 puzel users 64 Jun 24 10:36 0 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 1 -> /dev/pts/25
lrwx------ 1 puzel users 64 Jun 24 10:36 2 -> /dev/pts/25
lr-x------ 1 puzel users 64 Jun 24 10:36 3 -> anon_inode:inotify
lr-x------ 1 puzel users 64 Jun 24 10:36 4 -> anon_inode:inotify

Unless I misunderstood the concept, the following command should show you list of processes (their representation in /proc), sorted by number of inotify instances they use.

$ for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr

Finding the culprits

Via the comments below @markkcowan mentioned this:

$ find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -exec sh -c 'cat $(dirname {})/../cmdline; echo ""' \; 2>/dev/null

You are probably running out of inotify watches rather than instances. To find out who's creating a lot of watches:

  1. Enable tracing of watch adds:
$ echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable`
  1. Verify if tracing_on is s to 1:
$ cat /sys/kernel/debug/tracing/tracing_on
0
$ echo 1 > /sys/kernel/debug/tracing/tracing_on
  1. Restart the processes with inotify instances (determined as described in Petr Uzel's answer) that you suspect of creating a lot of watches; and
  2. Setup ftrace
$ cat /sys/kernel/debug/tracing/current_tracer
nop

$ cat /sys/kernel/debug/tracing/set_ftrace_filter
#### all functions enabled ####

$ echo function              > /sys/kernel/debug/tracing/current_tracer
$ echo SyS_inotify_add_watch > /sys/kernel/debug/tracing/set_ftrace_filter
  1. Read the file /sys/kernel/debug/tracing/trace to watch how many watches are created and by which processes.

When you're done, make sure to echo 0 into the enable file (and the tracing_on file if you had to enable that as well) to turn off tracing so you won't incur the performance hit of continuing to trace.

NOTE: In older versions of the Linux kernel the /sys endpoint used to be called tracing_enabled, however it's now called tracing_on. If you find you're on an older edition of the kernel change /sys/kernel/debug/tracing/tracing_on to /sys/kernel/debug/tracing/tracing_enabled.


As @Jonathan Kamens said, you are probably running out of watchers. I have a premade script, inotify-consumers, that lists the top offenders for you (a newer version also lists the username owning the process, see below):

$ time inotify-consumers  

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
    6688    27262  /home/dvlpr/apps/WebStorm-2018.3.4/WebStorm-183.5429.34/bin/fsnotifier64
     411    27581  node /home/dvlpr/dev/kiwi-frontend/node_modules/.bin/webpack --config config/webpack.dev.js
      79     1541  /usr/lib/gnome-settings-daemon/gsd-xsettings
      30     1664  /usr/lib/gvfs/gvfsd-trash --spawner :1.22 /org/gtk/gvfs/exec_spaw/0
      14     1630  /usr/bin/gnome-software --gapplication-service
    ....

    7489  WATCHERS TOTAL COUNT

real    0m0.099s
user    0m0.042s
sys 0m0.062s

Here you quickly see why the default limit of 8K watchers is too little on a development machine, as just WebStorm instance quickly maxes this when encountering a node_modules folder with thousands of folders. Add a webpack watcher to guarantee problems ...

Even though it was much faster than the other alternatives when I made it initially, Simon Matter added some speed enhancements for heavily loaded Big Iron Linux (hundreds of cores) that sped it up immensely, taking it down from ten minutes (!) to 15 seconds on his monster rig.

How to use

inotify-consumers --help To get it on your machine, just copy the contents of the script and put it somewhere in your $PATH, like /usr/local/bin. Alternatively, if you trust this stranger on the net, you can avoid copying it and pipe it into bash over http:

$ curl -s https://raw.githubusercontent.com/fatso83/dotfiles/master/utils/scripts/inotify-consumers | bash 

       INOTIFY
       WATCHER
        COUNT     PID USER     COMMAND
    --------------------------------------
        3044   3933 myuser node /usr/local/bin/tsserver
        2965   3941 myuser /usr/local/bin/node /home/myuser/.config/coc/extensions/node_modules/coc-tsserver/bin/tsserverForkStart /hom
         979   3954 myuser /usr/local/bin/node /home/myuser/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/li
           1   7473 myuser /usr/local/bin/node --no-warnings /home/myuser/dev/dotfiles/common-setup/vim/dotvim/plugged/coc.nvim/build/i
           1   3899 myuser /usr/local/bin/node --no-warnings /home/myuser/dev/dotfiles/common-setup/vim/dotvim/plugged/coc.nvim/build/i

        6990  WATCHERS TOTAL COUNT

How does it work?

For reference, the main content of the script is simply this (inspired by this answer)

find /proc/*/fd \
    -lname anon_inode:inotify \
    -printf '%hinfo/%f\n' 2>/dev/null \
    \
    | xargs grep -c '^inotify'  \
    | sort -n -t: -k2 -r

Changing the limits

In case you are wondering how to increase the limits

$ inotify-consumers --limits 

Current limits
-------------
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288


Changing settings permanently
-----------------------------
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p # re-read config

Tags:

Kernel

Fedora

Inotify

Recent Posts