HTTPS is over 50 times slower then HTTP

I had the same issue, with nearly identical response time differences between HTTP and HTTPS. Turns out the issue was as in the answer by @htmltiger: Apache2 was simply running out of worker processes.

This causes new requests to be queued until a worker becomes free and can process the next one [source]. I suppose the reason why this only affects HTTPS and not also HTTPS is that nearly all your traffic is over HTTP and Apache gives HTTP and HTTPS requests the same priority, taking one request from each queue in turn. So when the HTTPS queue is much longer, requests wait much longer. Indeed there are two queues, as the queue is simply the Linux TCP connection queue mechanism, and Linux provides one queue per port.

Diagnostics

If this is your problem, the following symptoms will apply:

• The best indicator: on your server, apachectl status shows that all allowable worker processes are running. This is the case when no dots . are shwon in the process scoreboard line, indicating no "Open slot with no current process" left. The line might look like this for example:

  KKKKKKRKKKRRCWKKKCCKWKKKKCRCKKKKKKKCKCKKKKWRKKKKWRWKKKKKKCWKKWKKK
  

• You see messages like this in your main Apache2 error log (/var/log/apache2/error.log, not domain specific ones):

  [mpm_prefork:error] [pid 4715] AH00161: server reached MaxRequestWorkers 
      setting, consider raising the MaxRequestWorkers setting
  

• There are many processes in your Apache backlog. According to this in-depth article, you can see this from the unacked: value in ss -lti '( sport = :https )' output. Depending on the version or configuration of ss, that value might be missing though.

• Most of the delay (say, 17 of 20 s) is shown in the Firefox Network Console, in the "Timings" tab for the initial URL requested, as "Blocking".

Solution

This assumes you use the prefork MPM server module in Apache. It's similar for the "event" and "worker" MPM modules though – details.

1. Edit /etc/apache2/mods-enabled/mpm_prefork.conf and increase the MaxRequestWorkers setting.

2. If you increase it beyond the default of 256, you also have to set ServerLimit to the same value to make your change effective.

3. Apply the changes: service apache2 reload

4. Make sure in the scoreboard output of apachectl status that the new MaxRequestWorkers setting is effective. It has to be equivalent to the length of the scoreboard line in characters.

5. If the setting is not effective yet, search in /etc/apache2 for old configuration directives (and their even older deprecated synonyms) that could overwrite your change:

   grep -R MaxRequestWorkers /etc/apache2/*
   grep -R MaxClients /etc/apache2/*
   

Differential Diagnoses

In case you see HTTPS being much slower than HTTP but not every single time in a series of page reloads (just on average), then you might have a variant of this fancy problem, with two Apache2 servers running on SSL port 443.