Apache/Linux server, DoS attack from own IP

First thing: find out where do these requests come from. It has to be a local process, nothing else is likely to be able to spoof a TCP handshake on a modern Linux platform (nothing, that is, that would then proceed to waste such a feat on requesting random images).

If there are recurrent URLs, you can shadow them behind a RewriteRule so that any such request will actually trigger a script. In the script you can run additional checks to see whether the request is legit (and you will then output the proper headers just as if it was the image the legit client expects), or if it is one of the bogus requests. Against the bogus request you can log e.g. the incoming port. Armed with that, you can query netstat and find out the process. You can also run ps and inspect all active processes in the instant of the bogus request.

I am quite sure that the culprit will prove to be Apache itself (I once had a "cache priming" script go rogue on me due to a vhost modification - I had forgotten putting the script in crontab - and got really weird symptoms, somewhat like yours, until it all came back to me; but your case feels different).

To further refine the scene while containing costs, you can add PID/TID to Apache's CustomLog. Then you will be able to cross-check the requests received from the Apache child gone rogue.

Another possibility is to determine exactly how these requests are made. If through Apache, this means fopen_wrappers, cURL, socket functions, or maybe shell utilities (these should both appear in ps output and result in a much more massive server overload, though). You can prepare a series of script that will:

• gracefully restart Apache without any changes
• " " , disabling temporarily one of those functions
• " " , re-enabling same

After verifying (just to be sure) that a restart does not fix the problem (if it did, it would be a quite different can of worms), you can proceed to temporarily disable - a couple dozen seconds each, no more - one function after another. Suppose that disabling curl results in the system immediate return to normal: then you could restrict investigations to scripts using cURL, and maybe even wrap the cURL function with a logging wrapper.

In case the guilty party turns out not to be Apache, still you will be able to determine what is doing this; then either reinstall the affected program (even if I find it unlikely for any random anomaly to turn a program into a repeat-HTTP-GET-requestor) or inspect its configuration, ancillary data files, scripts, and so on and so forth, looking for any difference from a known clean installation. Since I don't usually believe in gremlins, I expect for some difference to stand out in the end.

Unix (and Linux) has a wealth of tools for analysing stuff like this. My first stop would be to grab the output of netstat -nap e.g. on my local machine...

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
...
tcp        0      0 192.168.0.2:80              192.168.0.2:59875           ESTABLISHED 5281/httpd
...
tcp        0      0 192.168.0.2:59875           192.168.0.2:80              ESTABLISHED 32588/chrome
...

Here I can see than chrome (pid 32588) is connected to port 80 / httpd (pid 5281). Since this is a pre-fork installation of apache, I can get more information about the httpd process by logging %P or by looking in /proc/5281/fd (latter will probably require scripting to grab the data at the time of the request).

This will allow you identify the client process.

The most likely candidates are a badly configured proxy or buggy code.

If this were my server, I'd be running an strace on Apache. Running this on every child process in prefork mode can be quite disk intensive, especially when your server is already being overloaded. You do have to keep an eye on your disk space as well, because if it runs out, Apache stops serving requests.

Make sure you use a snaplength long enough to capture the entire request: -s 400 should do.

If Apache is making requests to itself, any GET string will appear in the strace dumps for two different PIDs: one that made the request and one that received it. In the one that made the request, you want to find the request that it received and was processing when it made the request to itself.

I normally do something like this:

for x in `ps -ef | grep apache | awk '{print $2}'`; do strace -s 2000 -p $x -o trace.$x & done

If you want to limit yourself to a subset of Apache children for performance reasons, add a head in there:

for x in `ps -ef | grep apache | head | awk '{print $2}'`; do strace -s 2000 -p $x -o trace.$x & done

But be aware that this makes it less likely for you to capture what's happening.

Make sure you have two SSH sessions open as all those backgrounded tasks can still write to your session. When you want to stop stracing, either restart Apache or run this in the other one:

 for x in `ps -ef | grep strace | awk '{print $2}'`; do kill $x; done

---

My gut feeling on this one is a "static" module written in PHP that pre-processes images (resizing them for instance) before sending them to the client and it does this with include($image). If $image happens to contain an image URL from your own site rather than a file path from the local filesystem, recursive requests are the result.

It could be using the curl() functions rather than include().