SSL slow. Establishing secure connection taking too long

I had the same issue and after a lot of digging I found that the problem was caused by the fact that I had mod_unique_id installed.

Further checking shows that the module is a requirement for mod_security. I did remove mod_security at first which did not make any change and only after removing the mod_unique_id module as well things started to fly.

Hope this helps.