Do browsers create new TCP connections for each HTTP requests?

In HTTP/0.9 (not used anymore), each request used a separate TCP connection, and the end of a response was signalled by closing the connection.

In HTTP/1.0, separate connections are still the official default. However, an unofficial but very widely supported "Connection: Keep-Alive" request header can be used to request a persistent connection if the server supports it.

In HTTP/1.1, persistent connections became the default, and the old single-request behavior has to be requested explicitly. Usually multiple (2–5) persistent connections are used.

(Optionally requests may be pipelined, though this turned out to be difficult to implement and creates more issues than it solves (head-of-line blocking, etc.), so nobody uses HTTP/1.x pipelining.)

HTTP/2 (aka SPDY) was specifically designed to multiplex many requests at the same time. It has a framing/packetization layer which allows responses to arrive in any order and even simultaneously.

Resources: Wikipedia article and RFC 2616 section 8.1.

There is "Connection: Keep-Alive" header. See enwiki for details.

You can use Wireshark to capture and analyse connections and see all headers.