Difference between ulimit, lsof, cat /proc/sys/fs/file-max

1. file-max is the maximum number of files that can be opened across the entire system. This is enforced at the kernel level.

2. The man page for lsof states that:

In the absence of any options, lsof lists all open files belonging to all active processes.

This is consistent with your observations, since the number of files as reported by lsof is well below the file-max setting.

3. Finally, ulimit is used to enforce resource limits at a user level. The parameter 'number of open files' is set at the user level, but is applied to each process started by that user. In this case, a single Kafka process can have up to 1024 file handles open (soft limit).

You can raise this limit on your own up to the hard limit, 4096. To raise the hard limit, root access is required.

If Kafka is running as a single process, you could find the number of files opened by that process by using lsof -p [PID].

Hope this clears things up.

That's a common mistake: to compare the results of a raw lsof call with supposed limit.

For the global limit (/proc/sys/fs/file-max) you should have a look at /proc/sys/fs/file-nr; the first value indicates what is used and the last value is the limit.

The OpenFile limit is for each process but it can be defined on a user; see command ulimit -Hn for user limits and see /etc/security/limits.conf for definitions. Generally applied with "app user" eg:"tomcat": set limit to 65000 to user tomcat that will apply on java process it runs.

If you want to check the limits applied on a process, get its PID and then:

cat /proc/${PID}/limits

If you want to check how many files are opened by a process, get its PID and then:

ls -1 /proc/${PID}/fd | wc -l (note for ls it's 'minus one', not to confuse with 'minus el')

If you want to know details with lsof but only for those file handles that count for the limit, try these:

lsof -p ${PID} | grep -P "^(\w+\s+){3}\d+\D+"

lsof -p ${PID} -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -a

Remark: the 'files' are files / pipe / tcp connections / etc.

Note that sometimes you'll probably need to be root or to use sudo to obtain correct result for the commands; without privilege, sometimes you don't see an error, you just get fewer results.

Finally, if you want to know what files on your filesystem are accessed by a process, have a look at:

lsof -p ${PID} | grep / | awk '{print $9}' | sort | uniq