How to set HttpWebRequest.Timeout for a large HTTP request in C#

There are two timeouts that plague us in processing a large file upload. HttpWebRequest.Timeout and HttpWebRequest.ReadWriteTimeout. We'll need to address both.

HttpWebRequest.ReadWriteTimeout

First, let's address HttpWebRequest.ReadWriteTimeout. We need to disable "write stream buffering".

httpRequest.AllowWriteStreamBuffering = false;

With this setting changed, your HttpWebRequest.ReadWriteTimeout values will magically work as you would like them to, you can leave them at the default value (5 minutes) or even decrease them. I use 60 seconds.

This problem comes about because, when uploading a large file, if the data is buffered by the .NET framework, your code will think the upload is finished when it's not, and call HttpWebRequest.GetResponse() too early, which will time out.

HttpWebRequest.Timeout

Now, let's address HttpWebRequest.Timeout.

Our second problem comes about because HttpWebRequest.Timeout applies to the entire upload. The upload process actually consists of three steps (here's a great article that was my primary reference):

1. A request to start the upload.
2. Writing of the bytes.
3. A request to finish the upload and get any response from the server.

If we have one timeout that applies to the whole upload, we need a large number to accomodate uploading large files, but we also face the problem that a legitimate timeout will take a long time to actually time out. This is not a good situation. Instead, we want a short time out (say 30 seconds) to apply to steps #1 and #3. We don't want an overall timeout on #2 at all, but we do want the upload to fail if bytes stop getting written for a period of time. Thankfully, we have already addressed #2 with HttpWebRequest.ReadWriteTimeout, we just need to fix the annoying behaviour of HttpWebRequest.Timeout. It turns out that the async versions of GetRequestStream and GetResponse do exactly what we need.

So you want this code:

public static class AsyncExtensions
{
    public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout)
    {
        return Task.Factory.StartNew(() =>
        {
            var b = task.Wait((int)timeout.TotalMilliseconds);
            if (b) return task.Result;
            throw new WebException("The operation has timed out", WebExceptionStatus.Timeout);
        });
    }
}

And instead of calling GetRequestStream and GetResponse we'll call the async versions:

var uploadStream = httpRequest.GetRequestStreamAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

And similarly for the response:

var response = (HttpWebResponse)httpRequest.GetResponseAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

That's all you'll need. Now your uploads will be far more reliable. You might wrap the whole upload in a retry loop for added certainty.