How can I tune the initial TCP retransmit timeout?

Solution 1:

Nope, you can't; it is hardcoded in the kernel. So change the kernel and recompile.

#define TCP_TIMEOUT_INIT ((unsigned)(3*HZ))     /* RFC 1122 initial RTO value   */

This is what you should get in your include/net/tcp.h.

But I can see someone provided a patch, even though never tried it myself

Solution 2:

The initial setting should not affect your overall performance much, as RTO self-adjusts to network conditions. If you do change RTO, you can set it to 1 sec (but no lower).

There is a discussion of this in RFC 1122:

        The following values SHOULD be used to initialize the
        estimation parameters for a new connection:

        (a)  RTT = 0 seconds.

        (b)  RTO = 3 seconds.  (The smoothed variance is to be
             initialized to the value that will result in this RTO).

        The recommended upper and lower bounds on the RTO are known
        to be inadequate on large internets.  The lower bound SHOULD
        be measured in fractions of a second (to accommodate high
        speed LANs) and the upper bound should be 2*MSL, i.e., 240
        seconds.

        DISCUSSION:
             Experience has shown that these initialization values
             are reasonable, and that in any case the Karn and
             Jacobson algorithms make TCP behavior reasonably
             insensitive to the initial parameter choices.

RFC 6298 is a proposed update (published June 2011) that says that RTO can be initialized to a lower value (but no lower than 1 sec), and contains an Appendix containing data that justifies 1 sec as a reasonable initial value.