Cannot set some HTTP headers when using System.Net.WebRequest

If you need the short and technical answer go right to the last section of the answer.

If you want to know better, read it all, and i hope you'll enjoy...

---

I countered this problem too today, and what i discovered today is that:

1. the above answers are true, as:

   1.1 it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again.

   1.2 Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist.

Thanks FOR and Jvenema for the leading guidelines...

2. But, What i found out, and that was the missing piece in the puzzle is that:

   2.1 The WebHeaderCollection class is generally accessed through WebRequest.Headers or WebResponse.Headers. Some common headers are considered restricted and are either exposed directly by the API (such as Content-Type) or protected by the system and cannot be changed.

The restricted headers are:

• Accept
• Connection
• Content-Length
• Content-Type
• Date
• Expect
• Host
• If-Modified-Since
• Range
• Referer
• Transfer-Encoding
• User-Agent
• Proxy-Connection

---

So, next time you are facing this exception and don't know how to solve this, remember that there are some restricted headers, and the solution is to modify their values using the appropriate property explicitly from the WebRequest/HttpWebRequest class.

---

Edit: (useful, from comments, comment by user Kaido)

Solution is to check if the header exists already or is restricted (WebHeaderCollection.IsRestricted(key)) before calling add

I ran into this problem with a custom web client. I think people may be getting confused because of multiple ways to do this. When using WebRequest.Create() you can cast to an HttpWebRequest and use the property to add or modify a header. When using a WebHeaderCollection you may use the .Add("referer","my_url").

Ex 1

WebClient client = new WebClient();
client.Headers.Add("referer", "http://stackoverflow.com");
client.Headers.Add("user-agent", "Mozilla/5.0");

Ex 2

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Referer = "http://stackoverflow.com";
request.UserAgent = "Mozilla/5.0";
response = (HttpWebResponse)request.GetResponse();

All the previous answers describe the problem without providing a solution. Here is an extension method which solves the problem by allowing you to set any header via its string name.

Usage

HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.SetRawHeader("content-type", "application/json");

Extension Class

public static class HttpWebRequestExtensions
{
    static string[] RestrictedHeaders = new string[] {
            "Accept",
            "Connection",
            "Content-Length",
            "Content-Type",
            "Date",
            "Expect",
            "Host",
            "If-Modified-Since",
            "Keep-Alive",
            "Proxy-Connection",
            "Range",
            "Referer",
            "Transfer-Encoding",
            "User-Agent"
    };

    static Dictionary<string, PropertyInfo> HeaderProperties = new Dictionary<string, PropertyInfo>(StringComparer.OrdinalIgnoreCase);

    static HttpWebRequestExtensions()
    {
        Type type = typeof(HttpWebRequest);
        foreach (string header in RestrictedHeaders)
        {
            string propertyName = header.Replace("-", "");
            PropertyInfo headerProperty = type.GetProperty(propertyName);
            HeaderProperties[header] = headerProperty;
        }
    }

    public static void SetRawHeader(this HttpWebRequest request, string name, string value)
    {
        if (HeaderProperties.ContainsKey(name))
        {
            PropertyInfo property = HeaderProperties[name];
            if (property.PropertyType == typeof(DateTime))
                property.SetValue(request, DateTime.Parse(value), null);
            else if (property.PropertyType == typeof(bool))
                property.SetValue(request, Boolean.Parse(value), null);
            else if (property.PropertyType == typeof(long))
                property.SetValue(request, Int64.Parse(value), null);
            else
                property.SetValue(request, value, null);
        }
        else
        {
            request.Headers[name] = value;
        }
    }
}

Scenarios

I wrote a wrapper for HttpWebRequest and didn't want to expose all 13 restricted headers as properties in my wrapper. Instead I wanted to use a simple Dictionary<string, string>.

Another example is an HTTP proxy where you need to take headers in a request and forward them to the recipient.

There are a lot of other scenarios where its just not practical or possible to use properties. Forcing the user to set the header via a property is a very inflexible design which is why reflection is needed. The up-side is that the reflection is abstracted away, it's still fast (.001 second in my tests), and as an extension method feels natural.

Notes

Header names are case insensitive per the RFC, http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2