How to circumvent "Too many open files" in debian

It is Important to know that there are two kinds of limits:

• A hard limit is configurable by root only. This is the highest possible value (limit) for the soft limit.
• A soft limit can be set by an ordinary user. This is the actual limit in effect.

Solution for a single session

In the shell set the soft limit:

ulimit -Sn 2048

This example will raise the actual limit to 2048 but the command will succeed only if the hard limit (check: ulimit -Hn) is the same or higher. If you need higher values, raise the hard limit using one of the methods below. The limits are set per process and they are inherited by newly spawned processes, so anything you run after this command in the same shell will have the new limits.

Changing hard limit in a single session

This is not easy because only root can change a hard limit and after switching to root you have to switch back to the original user. Here is the solution with sudo:

sudo sh -c "ulimit -Hn 9000 ; exec su \"$USER\""

System-wide solution

In Debian and many other systems using pam_limits you can set the system-wide limits in /etc/security/limits.conf and in files in /etc/security/limits.d. The conf file contains description. Example lines:

@webadmins       hard     nofile     16384
@webadmins       soft     nofile      8192

This will set the hard limit and default soft limit for users in group webadmins after login.

Other limits

The hard limit value is limited by global limit of open file descriptors value in /proc/sys/fs/file-max which is pretty high by default in modern Linux distributions. This value is limited by NR_OPEN value used during kernel compilation.

Is there not a better solution?

Maybe you could check if all the *log files you feed to tail -f are really active files which need to be monitored. It is possible that some of them are already closed for logging and you can just open a smaller number of files.