What is the meaning of "AH00485: scoreboard is full, not at MaxRequestWorkers"?
Solution 1:
We had the same problem on Apache 2.4.6. After monitoring the server and adjusting the setting for several hours it appears to us that Apache may have a bug. What appears to happen is that the server processes occasionally goes into the G state (Gracefully finishing) and restarts to accept new requests, that's normal. What is not normal is that for some reason this can take up to a few minutes to restart. If you only have a few server process running and they all go into the G state at the same time then your scoreboard fills up and you won't be able to server any more requests.
What we did was increase the number of servers so there is a less of a chance that they will all go into the G state at the same time. Also make sure you allocate at least 25 threads (MaxRequestWorkers) for each server process because that appears to be the default (i.e. if 5 Servers x 25 ThreadsPerChild = 125 MaxRequestWorkers). You can change ThreadsPerChild if you like, we left it at default. If you don't allocate enough threads the additional servers will not start. We left MinSpareThreads at the default value which is 25 and the default for MaxSpareThreads which is 75. If you do modify these settings, the value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild. Also MaxRequestWorkers must be equal to or less than the ServerLimit.
Here is what worked for us but it might not be the best configuration for you.
StartServers 3
MinSpareServers 5
MaxSpareServers 10
ServerLimit 250
MaxRequestWorkers 250
MaxConnectionsPerChild 1000
KeepAlive Off
Edit: This is a confirmed bug in httpd's mpm_event module which might not be fixable through configuration.
The linked bugtracker entry has a presumed patch and more discussion about how to fix this until a new version of the event module is officially released.
Solution 2:
Seeing the same issue.
Apache 2.4.7-1ubuntu4.4 on Ubuntu 14.04
Server Version: Apache/2.4.7 (Ubuntu)
Server MPM: event
Server Built: Mar 10 2015 13:05:59
We particularly can cause this behaviour by reloading apache.
What we then see, are a couple of old processes that don't stop:
root 28192 0.0 0.8 103772 8648 ? Ss Mar16 0:03 /usr/sbin/apache2 -k start
www-data 2530 0.3 2.1 865188 21516 ? Sl 06:26 0:54 \_ /usr/sbin/apache2 -k start
www-data 2531 0.2 2.1 865436 21892 ? Sl 06:26 0:51 \_ /usr/sbin/apache2 -k start
www-data 3299 0.3 2.0 864140 20628 ? Sl 06:46 0:51 \_ /usr/sbin/apache2 -k start
www-data 7305 0.3 2.1 865100 21504 ? Sl 08:36 0:37 \_ /usr/sbin/apache2 -k start
www-data 11952 0.2 1.8 863004 19268 ? Sl 10:46 0:06 \_ /usr/sbin/apache2 -k start
www-data 13284 0.0 0.6 103772 6692 ? S 11:18 0:00 \_ /usr/sbin/apache2 -k start
www-data 13553 2.1 2.0 866156 21248 ? Sl 11:23 0:01 \_ /usr/sbin/apache2 -k start
Notice the 'older' and 'newer' PID's and start times. ^^
PID Connections Threads Async connections
total accepting busy idle writing keep-alive closing
7305 14 no 0 0 0 0 0
2530 13 no 0 0 0 0 0
3299 7 no 0 0 0 0 0
13553 65 no 17 8 0 25 25
2531 15 no 0 0 0 0 0
11952 10 no 0 0 0 0 0
Sum 124 17 8 0 25 25
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGW_WWWW__W_W_W_WWWWWWW__WWGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGG