fork: retry: Resource temporarily unavailable

Another possibility is too many threads. We just ran into this error message when running a test harness against an app that uses a thread pool. We used

watch -n 5 -d "ps -eL <java_pid> | wc -l"

to watch the ongoing count of Linux native threads running within the given Java process ID. After this hit about 1,000 (for us--YMMV), we started getting the error message you mention.

This is commonly caused by running out of file descriptors.

There is the systems total file descriptor limit, what do you get from the command:

sysctl fs.file-nr

This returns counts of file descriptors:

<in_use> <unused_but_allocated> <maximum>

To find out what a users file descriptor limit is run the commands:

sudo su - <username>
ulimit -Hn

To find out how many file descriptors are in use by a user run the command:

sudo lsof -u <username> 2>/dev/null | wc -l

So now if you are having a system file descriptor limit issue you will need to edit your /etc/sysctl.conf file and add, or modify it it already exists, a line with fs.file-max and set it to a value large enough to deal with the number of file descriptors you need and reboot.

fs.file-max = 204708