How to get the file size from http headers

Note that not every server accepts HTTP HEAD requests. One alternative approach to get the file size is to make an HTTP GET call to the server requesting only a portion of the file to keep the response small and retrieve the file size from the metadata that is returned as part of the response content header.

The standard System.Net.Http.HttpClient can be used to accomplish this. The partial content is requested by setting a byte range on the request message header as:

    request.Headers.Range = new RangeHeaderValue(startByte, endByte)

The server responds with a message containing the requested range as well as the entire file size. This information is returned in the response content header (response.Content.Header) with the key "Content-Range".

Here's an example of the content range in the response message content header:

    {
       "Key": "Content-Range",
       "Value": [
         "bytes 0-15/2328372"
       ]
    }

In this example the header value implies the response contains bytes 0 to 15 (i.e., 16 bytes total) and the file is 2,328,372 bytes in its entirety.

Here's a sample implementation of this method:

public static class HttpClientExtensions
{
    public static async Task<long> GetContentSizeAsync(this System.Net.Http.HttpClient client, string url)
    {
        using (var request = new System.Net.Http.HttpRequestMessage(System.Net.Http.HttpMethod.Get, url))
        {
            // In order to keep the response as small as possible, set the requested byte range to [0,0] (i.e., only the first byte)
            request.Headers.Range = new System.Net.Http.Headers.RangeHeaderValue(from: 0, to: 0);

            using (var response = await client.SendAsync(request))
            {
                response.EnsureSuccessStatusCode();

                if (response.StatusCode != System.Net.HttpStatusCode.PartialContent) 
                    throw new System.Net.WebException($"expected partial content response ({System.Net.HttpStatusCode.PartialContent}), instead received: {response.StatusCode}");

                var contentRange = response.Content.Headers.GetValues(@"Content-Range").Single();
                var lengthString = System.Text.RegularExpressions.Regex.Match(contentRange, @"(?<=^bytes\s[0-9]+\-[0-9]+/)[0-9]+$").Value;
                return long.Parse(lengthString);
            }
        }
    }
}

Yes, assuming the HTTP server you're talking to supports/allows this:

public long GetFileSize(string url)
{
    long result = -1;

    System.Net.WebRequest req = System.Net.WebRequest.Create(url);
    req.Method = "HEAD";
    using (System.Net.WebResponse resp = req.GetResponse())
    {
        if (long.TryParse(resp.Headers.Get("Content-Length"), out long ContentLength))
        {
            result = ContentLength;
        }
    }

    return result;
}

If using the HEAD method is not allowed, or the Content-Length header is not present in the server reply, the only way to determine the size of the content on the server is to download it. Since this is not particularly reliable, most servers will include this information.

WebClient webClient = new WebClient();
webClient.OpenRead("http://stackoverflow.com/robots.txt");
long totalSizeBytes= Convert.ToInt64(webClient.ResponseHeaders["Content-Length"]);
Console.WriteLine((totalSizeBytes));

Can this be done with HTTP headers?

Yes, this is the way to go. If the information is provided, it's in the header as the Content-Length. Note, however, that this is not necessarily the case.

Downloading only the header can be done using a HEAD request instead of GET. Maybe the following code helps:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://example.com/");
req.Method = "HEAD";
long len;
using(HttpWebResponse resp = (HttpWebResponse)(req.GetResponse()))
{
    len = resp.ContentLength;
}

Notice the property for the content length on the HttpWebResponse object – no need to parse the Content-Length header manually.