How can lsof report a higher number of open files than what ulimit says should be allowed?

From the ulimit builtins man page

The ulimit builtin provides control over the resources available to the shell 
and to processes started by it on systems that allow such control.

Your lsof command lists all of the open files for all processes for all users on the system. You are not comparing like with like.

That's a common mistake to compare results of raw lsof call with supposed limit.

For the global limit (/proc/sys/fs/file-max) you should have a look at /proc/sys/fs/file-nr -> the first value indicates what is used and the last value is the limit

The OpenFile limit is for each process, but it is defined on a user, see command ulimit -Hn for user limits and see /etc/security/limits.conf for definitions. Generally applied with "app user" eg:"tomcat": set limit to 65000 to user tomcat that will apply on java process it runs.

If you want to check limit applied on a process, get its PID and then do: cat /proc/${PID}/limits If you want to check how many files are opened by a process, get its PID and then od: ls -1 /proc/{PID}/fd | wc -l (note: for ls it's 'minus one', not to confond with 'minus el')

If you want to know details with lsof but only for those file handlers that count for the limit, have a try with thoses commands:

• lsof -p ${PID} | grep -P "^(\w+\s+){3}\d+\D+"
• lsof -p ${PID} -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -a

Remark : the 'files' are files / pipe / tcp connections / etc.

Note that sometimes you'll probably need to be root or to use sudo to obtain correct result for the commands. Without privilege sometimes you don't have error, just less results.

Finally, if you want to know what 'files' on your filesystem are accessed by a process, have a look at: lsof -p {PID} | grep / | awk '{print $9}' | sort | uniq

have fun !

Although this is old, I wanted to ask the same question... the answer is not satisfactory in my case as this is what happens:

$ sudo su tomcat -c "ulimit -n"
1024
$ lsof -u tomcat
3967

I'm not entirely sure why this happened. I guess that open files from sub process are not accounted for in the parent one.