How can I keep Apache from falling over?

I normally setup the MaxRequestsPerChild to be in the thousands - usually, nearer 10,000.

You say that you have "the recommended PHP caching" - but do you have APC installed? Finally, how many users do you see hitting the website at the same time. If you have Apache extended stats, you will be able to see how many of the Apache processes are actually in the Running state at a time.

800 APC file hits per second, and another 200 user-cache is a lot. If that is a dual or quad-core, I'd expect it to be keeping up OK though. If the database is genuinely keeping up, then getting a bigger machine - and more CPUs, may be the best thing for it, at least right now.

Magento and Zend Framework are quite CPU-heavy, as you noticed. The best way to avoid the CPU load is simply but rendering any content only once, until it changes. Most parts of your catalog don't change that often, and often only the shopping cart block on your page, or the 'most popular items' block are the only dynamic parts.

I would suggest putting a Varnish cache in front of Apache. This gives you high-performance page-caching that can seriously offload your LAMP stack. We recently survived a very public launch of a website thanks to Varnish and I was seriously impressed by the speed and low cpu-load. Varnish is free, and flexible enough to cache entire pages, or cache only the relatively static parts and include the cart dynamically.

However, Varnish will not cache much on a default Magento installation, since there's a lot of per-user dynamic content, cookies, etc. A Magento module such as 'PageCache powered by Varnish' modifies Magento to work well with Varnish. It also provides a Varnish configuration file that matches the Magento setup. These two together make for a very efficient setup. It's a commercial module, but much more affordable than a more powerful server would be.

The parts your offloading to a CDN or Nginx are not your real problem, although it does help. Even Apache can handle quite a number of static requests. You need to cache the stuff that takes effort to generate again and again, i.e. your dynamic parts.

Your average load is entirely too high for a dual core VPS. 8 should be the max.

I've had good success with using mod_pagespeed and event MPM for Magento. I would recommending switching to using event MPM, and installing mod_pagespeed.

More info about Event MPM: Apache event MPM documentation

And mod_pagespeed: Google Code: mod_pagespeed

If you continue to have load issues even after making the above changes, you may want to consider switching to a different, better VPS plan.