Delay and low speed in loading iframe content at huge visit

At this particular point in the size of your application I would focus on two specific improvements to your infrastructure and code logic.

1. Decouple the processes needed to serve banner assets to the end user from the processes needed to store ad impressions.
2. Implement a horizontally scalable infrastructure for receiving requests.

1. Decoupling serving banners from storing impressions

By separating these two concerns you will be able to provide more consistent performance for the customers of your ppc service. If you aren't already, using a CDN or otherwise offloading the demand of serving the images themselves from your severs can help improve response time considerably.

Another area that will give you large gains is separating the serving of banner code from the processes that store the impression data to disk. There are a number of ways to do this, but a successful solution I have had experience with is utilizing ActiveMQ (http://activemq.apache.org/) or a similar queuing system. A queuing system will help balance your impression storage load over time by storing impression data in memory and sending off those data points at a consistent rate to consumer (aka worker) processes that can store that data into a DB or other storage medium. This allows the workload of actually storing impressions on disk to be separated from the process of serving ads. You can also set up multiple processes to consume the queued up jobs, which leads to the second area of improvement.

2. Horizontally scalable infrastructure

Building a horizontally scalable solution basically means that instead of needing to increase the size and power of a single server, you can just add additional smaller servers that will evenly share the workload of the system demands. This has multiple advantages, one of which being it is easier (and usually cheaper) to add a few more small servers to a pool rather than upgrading one big server to be larger and more powerful. It also had the advantage of being more robust in the event that a server fails.

In this case I think a good solution would be to have one server or process acting as a router, which will just load balance requests by sending them off to different servers that are doing the actual processing of the request. There are a lot of good resources about building routing or load balancing scripts in PHP out there on the internet, but basically you will receive requests at one endpoint, and then send that request to a different server to actually be fulfilled. If you build a dynamic list of servers that are ready to receive requests then you can easily increase the number of servers fulfilling requests when your start to see unacceptable performance. This will also give you the benefit of being able to easily remove a server from the list if it goes down, and then any traffic would just get routed to a different server that was still up.

If you haven't already, it would be good to look into lighttpd (http://www.lighttpd.net/) or nginx (https://www.nginx.com/) as alternatives to Apache which are built to be able to handle large volumes of requests with less overhead. These would be especially well suited to handling the requests on your router server.

Once you have horizontal scaling set up for requests, it would be fairly simply to set up horizontal scaling for storage servers as well. You can easily do this by modding an ID by the number of servers in the pool to determine where to send the request.

$serverNumber = $adID % $availableServers;

Summary

Although you can definitely see good performance improvements by optimizing storage methods and server tuning, at some point in a large application you will want to be able to add additional servers to get the job done. I think with the above steps you will be in a very good place to scale your application smoothly as it grows in size.

You might be able to save on both performance and development hours by leveraging your existing Apache access logging to act as your impression tracker, and completely turn off your custom database/flat-file read/write code.

Set up a new Apache virtual host, and point it to a system webroot directory that you will make solely responsible for serving up your ad-generation PHP script. Within this one virtual host configuration, you can set up a system access log which is 100% exclusive to serving your ads; Apache will take care of appending to this log upon every visit (impression) and rotating/archiving the log as needed. You can point to wherever you would like to store the log as well as what kind of server/environment/referrer/user-agent data gets stored to it.

Then, via cron or a daemon, you can run whatever back-end log analyzer you would like, gather statistics into a database, crunch your numbers, etc, without coupling any of this heavier lifting to live web requests.

I also found on Stack Overflow the suggestion to use this technique with lighttpd instead of Apache to further stretch your server resources.

Further reading:

• Clever Impression Tracking Technique

• Apache: Log Files

• lighttpd

With 3k requests a minute, and with hope of growth, you need to start utilizing big data architecture and tools.

Here are some broad picture highlights to consider:

• Use a seperate CDN to store and serve the images.
• Use mapReduce software to store the data, such as hadoop.
• Get servers which are distributed, as opposed to one huge server.
• Add a load balancing server.
• Turn off all php features & extensions you don't need.