Getting 408 errors on our logs with no request or user agent
Solution 1:
Are you by any chance running your web servers in Amazon behind an Elastic Load Balancer?
It seems they generate a lot of 408 responses due to their health checks.
Some of the solutions from that forum thread:
RequestReadTimeout header=0 body=0This disables the 408 responses if a request times out.
- Change the ELB health check to a different port.
Disable logging for the ELB IP addresses with:
SetEnvIf Remote_Addr "10\.0\.0\.5" exclude_from_log CustomLog logs/access_log common env=!exclude_from_log
And from this blog post:
- Adjust your request timeout to be 60 or above.
Solution 2:
There are already quite a few good answers here, but I would like to hazard one additional note that hasn't been specifically addressed. As many of the previous commenters already mentioned, 408 indicates a timeout; and there are quite an array of circumstances in which timeouts occur on a web server.
With that said, 408 errors can be generated in a variety of cases where your server is being scanned for exploits. Clients in such cases rarely present a user agent and often end connections abruptly, resulting in an abortive shutdown for that connection that can generate a 408 error.
For example, let's say that I am a dastardly hacker who is scanning the internet for computers that are still vulnerable to the POODLE vulnerability. As such, I have written a script that opens connections to large chunks of IP addresses to find servers that will accept SSL version 3 - later on I will use that list to scan for the POODLE exploit specifically. All that this first script does is establish a connection using openssl to check for SSLv3, like this:
openssl s_client -connect [IP]:443 -ssl3
This command will, in many configurations of Apache, result in a 408 message exactly as you have described. Performing this command on two of my own servers resulted in this entry to the access log:
<remote IP address> - - [04/Nov/2015:08:09:33 -0500] "-" 408 - "-" "-"
I wanted to make this clear as even in a situation where OP was not using any form of load balancing, 408 errors can occur in a variety of circumstances - some malicious, some indicating trouble with the client, and some indicating trouble with the server. (I noticed in the log provided by OP that a local IP was indicated as the remote IP, but OP did not specifically mention the use of a load balancer so I was not sure if OP had simply used a non-routable IP for the purposes of demonstration, as he did with the URL)
Anyway, even though my post if obviously way too late in the day to help the OP hopefully it might help others who arrive here looking for a solution to all of those damn timeout errors.
Solution 3:
Something is connecting to the port and then never sending data. HTTP 408 is a "timeout" error. There's a good writeup here: http://www.checkupdown.com/status/E408.html
Solution 4:
There are various reasons for a 408 Timeout. But lets start from a premise that everything's fine then at some point these 408's start appearing in your access log - i.e. 408 0 "-" "-".
Like many point out on the net, a 408 represents a connection is made but no request is sent in the appropriate time scale, therefore the server drops the connection with a 408. One arrogant individual actually responded to someone asking for help on this issue with - "What part of Timeout do you not understand".
I think that is very much a novice answer and demonstrates a total lack of understanding of how some security methods function with web-server software.
So back to the beginning, why am I seeing all these 408's. One thing you will have in common with the rest of us managing a server is the vast amount of attacks you receive on a daily basis. Now, what do you do about these? Well: you employ your chosen security methods to deal with them, this is what changes.
Let’s take a very easy example, Drop an IP address. Included in an iptabes file (rules.v4) you have "-A ufw-user-input -s 37.58.64.228 -j DROP". So along comes 37.58.64.228 the firewall recognises the IP and drops the connection. In a lot of configurations you wouldn't even know it's knocked on the door.
Now lets take a more advanced example, Drop the connection based on some criteria. Included in an iptabes file (rules.v4) you have "-A INPUT -p tcp -m tcp --dport 80 -m string --string "cgi" --algo bm --to 1000 -j DROP". This is different because in this iptable rule we are saying look at the first 1000 bytes of the request string and see if you can find a sub-string of "cgi" and if you do find that sub-string then don't go any further, just drop the connection.
Here the security method is good, but as far as your logs are concerned misleading. The 408 0 "-" "-" generated is the best apache can do under these circumstances. The connection was made and the request had to be accepted up to a point in order to apply the string comparison rule which ultimately results in a 408 because your rule met the criteria for the connection to be dropped. So, our little novice darlings couldn't be more wrong if they tried. A connection was made and a request was received (you just won't have visibility of it under these circumstances). Although a 408 is generated it is not a 'Timeout'; your server simply dropped the connection after the request was made in association with your firewall rule. There are many other rules, which would create the same 408 situation. Don't accept too literally the Server Error Code 408 explanation - 'Request Timed Out'.
Ideally there would be another Apache generated Error Code, for example - '499' which would mean 'Server Read Your Request And Decided It Just Couldn't Be Bothered Entertaining You - Sod Off HaHa'.
With the latest web-server software you can practically exclude DOS attacks and the new gene of browsers incorporating predictive capabilities don't cause this problem as some have suggested.
In short, the 408 is generated because the server did not respond to the request, so as far as the client is concerned the connection timed out, when in reality the server read the request but dropped the connection for reasons other than a timeout waiting for a request.
Solution 5:
We had this very problem, and puzzled over it for quite a while. The best solution we came up with was suggested by AWS support's ELB team. It essentially hinges on ensuring that your httpd server's timeout settings are all greater than your ELB idle timeout setting (which defaults to 60 seconds).
- Ensure that your apache
Timeoutdirective value is double theidle timeoutsetting of your ELB. - Turn on the
KeepAlivefeature, ensure thatMaxKeepAliveRequestsis very large (either 0 for infinite or very high, like 2000), and thatKeepAliveTimeoutis greater than your ELB'sidle timeout.
We found that the KeepAlive (and associated settings) setting specifically reduced the quantity of 408s to effectively 0 (we see a few, but very few).