Why is Go HTTPS Client not reusing connections?

There are interesting improvements in http.Transport:

// DisableKeepAlives, if true, disables HTTP keep-alives and
// will only use the connection to the server for a single
// HTTP request.
//
// This is unrelated to the similarly named TCP keep-alives.
DisableKeepAlives bool

// ...

// MaxIdleConns controls the maximum number of idle (keep-alive)
// connections across all hosts. Zero means no limit.
MaxIdleConns int // Go 1.7

// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
// (keep-alive) connections to keep per-host. If zero,
// DefaultMaxIdleConnsPerHost is used.
MaxIdleConnsPerHost int

// MaxConnsPerHost optionally limits the total number of
// connections per host, including connections in the dialing,
// active, and idle states. On limit violation, dials will block.
//
// Zero means no limit.
MaxConnsPerHost int // Go 1.11

Based on further clarification from the OP. The default client does reuse connections.

Be sure to close the response.

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Additionally, I've found that I also needed to read until the response was complete before calling Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

To ensure http.Client connection reuse be sure to do two things:

• Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
• Call Body.Close()

---

Old answer, useful for rate limiting, but not what the OP was after:

I don't think setting max connections is possible via the golang 1.1 http APIs. This means you can shoot yourself in the foot with tons of TCP connections (until you run out of file descriptors or whatever) if you aren't careful.

That said, you could limit the rate at which you call the go routine for a particular host (and therefore outbound requests and connections) via time.Tick.

For example:

import "time"

requests_per_second := 5
throttle := time.Tick(1000000000 / requests_per_second)

for i := 0; i < 16; i += 1 {
  <-throttle  
  go serveQueue()
}