One-time mystery redirect from one of our pages
The first place I would be looking to find what my server has done is in my logs. A redirect issued as a header would show there (but not a meta or javascript redirect).
At least this would tell me which code to audit.
have I missed anything obvious to check?
It seems to me like you've done quite a bit of analysis local to you, so you've probably missed nothing obvious, but there are some subtle things you might have missed. As supplement to this answer I'll explain multiple attacks that could be so subtle and virtually out of your control that it's possible for any insecure website that uses a hostname and/or client that uses an insecure DNS and HTTPS configuration (i.e. any that use plain old HTTP and don't rely explicitly upon DNSSEC) to fall victim, and the attackers can sometimes virtually infeasible to actively shut down. I'll also describe how you can prevent these subtle attacks, in case you missed the subtle hints in this paragraph.
Could there be an Apache / server level compromise doing this, for example?
I guess you could call an insecure configuration a compromise, especially if it's intentionally configured to be insecure there by an attacker. The most dangerous of these misconfigurations could be present in the form of rogue administration accounts on a server, rogue software installation, rogue certificate authorities, so on and so forth. Antivirus software may not detect all compromises (misconfigurations), and it's important to consider them in light of a security regime which consists of multiple components, not just antivirus software.
You should keep all of your software up-to-date, particularly Apache, your OS and particularly web browsers as there are attacks which target specific versions of Apache, your OS and web browsers, and I'd suggest that all of those kinds of attacks are commonly used. The ones described here, however, are less likely to be commonly observed, owing to their risk factors or the level of skill required to reproduce them in ratio to the amount of maturity one must possess to establish that skill, and much less likely to be detected by antivirus software owing to their existence as flaws in configuration rather than as malware. For some reason, there's a distinction between these two concepts. I think there shouldn't be.
Simply put, the internet is held in balance by the thin layer of ice which is the fact that the mature, highly experienced users of the internet (the top 1%, if you like) haven't decided to wage war against everyone on the planet. Sure, there may be some Russian and Chinese hackers who ruin it, but those malicious IP subnets can be blocked by firewalls and DNSBLs, their scans crippled by tarpits, so on and so forth.
The first attack to consider is as simple as a service provider filtering their customers traffic. In doing so, they could quite easily set up a phishing site over the top of yours, or inject some Javascript which redirects the user. Such an attack would be blatantly obvious, and indefensible in court, nonetheless you might still observe this.
This could occur anywhere between your server and your client, even on your own (or their own) routers as we'll inspect later. For now, we'll focus on the routers that exist between your network and your clients network which neither of you own. This could include proxy servers and nameservers.
Typical ISPs run at least a caching DNS resolver for the clients to reduce congestion along the up-stream network, and that is itself a form of filtering. Some might even run proxy software which compresses media files and/or caches static HTML pages to reduce bandwidth for clients. Each of these caches has the potential to be tampered with, internally or externally.
Internally, a malicious employee at an ISP could likely corrupt the local DNS cache to give the client a different IP for your hostname, or corrupt a proxy to send a different page (or perhaps a different version of jquery). That'd be very blatant, incredibly stupid attack as the risk of being caught in this highly illegal attack increases as attacks are repeated. Nonetheless, if you're an American citizen, best of luck getting a maliciously hosted public proxy hosted in China taken down...
Besides, without some source code for the page allegedly hosted by you that your client ended up on, there's no way you could prove that an upstream provider somewhere between your client and your server has interfered with traffic. You need to find that proof if you want to find the source, or solve the problem if you want to solve the problem. That source code is just one source of information you'd need to obtain to track down the culprit.
When a browser is sent a signed page from a server, it will contact some certificate authority to ensure the certificate that signed the page legitimately came from the server that it's apparently from. The responses sent by a certificate authority are themselves signed with keys which are generally hard-configured into your OS/network configuration/browser and thus fraudulently forging such a response (which we'll cover later with respect to other attacks) would be even less practical than somehow encouraging the end-users to overtly install a rogue certificate authority signature (like convincing them to install malicious software). There's no doubt that a hardened SSL configuration at both ends (server and end user) will prevent all of the attacks I describe here.
As it turns out, even home routers nowadays have their own caching resolvers, once again to reduce traffic congestion on the up-stream network. Thus, a malicious machine on the same private network might have no problems infecting your router and corrupting its DNS caching resolver.
Cue the cellular attacks: - A typical consumer-grade cell-phone internet user is technically also on a private network. Indeed, the cell router may be misconfigured to allow clients to attack other clients. The routers are 3G/HSPDA+ enabled, introducing the potential for numerous electromagnetic attacks which tend to be encrypted, but I suppose if a client enables roaming they might not necessarily know which towers they're connecting to or who runs those "towers", leading to a potential for MITM attacks which are similar to some other wireless network attacks. - The typical cell-phone client who has external access via some other provider can launch another style of attack against all other users of that mobile internet carrier within the local area (could be as vast as state region) and... any insecure website... could be yours, could be all, might be effected. What's important is that the content be unencrypted end-to-end. This client has everything they need to trigger the ISPs caching resolver to send a request for a zone (you can basically think of this as a domain, if you're uninitiated), and to forge thousands of fake responses for the request which will all beat any legitimate response to the target.
It's important to realise that a secure implementation of signed, domain-verified encryption such as a hardened HTTPS configuration will cripple all of the attacks I describe here, as I previously explained.
Attacks against DNS themselves aren't few and far between. There is a technique known as "bitsquatting", where-by a nefarious domain is registered which differs in just one bit from the target domain. For example, an attacker targeting an insecure HTTP configuration hosted by stackexchange.com might register the domain staciexchange.com. Notice the difference between k and i is only one bit. This targets any routers which might have RAM errors that occasionally flip that bit.
Nonetheless, a user connecting via a hardened HTTPS configuration will almost certainly be warned about invalid certificates if the attacker can't work out what their targets private key is in order to make the content appear legitimate. This would be present as a clue from your users report, if he/she said "The browser said the certificate was invalid", for example.
Those last few attacks make it really difficult to blame a server, client or router anywhere between. To prove who (perhaps inadvertently) tampered with the traffic, you'll need to call administrators for each router between yourself and the client and see what they have in their caches. This introduces extra higher-level challenges in your quest to track down the potential issue. You may be quickly shut down by any foreign ISPs or authority who you approach to track this down. They may speak a different language (such as Russian or Chinese), they may not have laws that criminalise these behaviours, they may behave in a hostile manner towards you because of your race and they may not want to admit insecurities in their own network.
However, there are things you can do to protect yourself from this specific attack, and as they say prevention is better than cure. Most significantly, you should try to at least gracefully upgrade users to a secure SSL connection. Additionally, if you do happen to get a call from a client and while you're on the phone they can repeatedly reproduce the issue, consider instructing them to take steps such as clearing the local DNS, proxy, and browser caches, one at a time, checking to see if the symptoms disappear each time. This will tell you whether the clients network (not just one computer) is at fault or not. Additionally, if you're not already using a DNSSEC-enabled nameserver for your zone, you probably should consider migrating to one, as that'll eliminate similar attacks for other non-HTTP services.
For the meantime, I'm going to assume HTTP is the service you want to harden, and since you can't guarantee users will harden their DNS configuration you should rely upon browsers and OSes hardening their HTTPS configurations instead, and thus try to upgrade your users connections to HTTPS where possible.
Regarding the clients computer having no antivirus on OS X, this is not enough information to determine whether the clients network is to blame. OS X isn't exactly insecure; sure, they have the occasional vulnerability, but those tend to get patched as with any OS. Where clients go wrong here is when they don't apply the automatic updates or patches necessary to keep their system secure, or they install rogue software themselves. In this case, default Apple configurations nowadays use a similar certificate authority technique to ensure binaries are signed by Apple, or else pop up a warning to the user (or flat out refuse to execute the program). Apple can, of course, revoke the legitimacy of a signature at their own discretion, and they rely largely upon this to protect their users against malware. It's a good system which package managers on Linux tend to also follow.
What's scariest of all is that there appears to be no antivirus software available to scan the software that executes on your printer. Thus, some printers might be capable of attacking the users of the networks they're installed on, or even external networks.
In summary, these are the things you might want to consider:
- Keep your computer software up-to-date, including your OS, any antivirus software which might or might not be necessary, and particularly any browser software. This goes for all internet users, yourself included, though for you there are some pieces of software you might want to consider: tarpits and DNSBL integration software being a few.
- Be wary of browser extensions, as these may themselves tamper with pages or introduce vulnerabilities.
- Host your zone on a DNSSEC-enabled nameserver. Just do it, mate. Don't ummm and arrr about it. This is like SPF for email servers; if you don't do it, you'll eventually become a victim of an attack.
- Configure your servers to gracefully upgrade your clients to SSL, at the very least.