Multithreading a large number of web requests in c#

You might create too many connections, thus using up all the local ports you can use. There's a timeout period for when a port can be reused after you close it. WebRequest hides all the low level socket handling for you, but I am guessing it eventually runs out of ports, or tries to (re)bind to a socket already in a TIME_WAIT state.

You should make sure you read the response stream, even if you don't care about the response. This should help not producing too many lingering connections.

WebResponse response = request.GetResponse();
new StreamReader(response.GetResponseStream()).ReadToEnd(); 

I'll paste some relevant info from here:

When a connection is closed, on the side that is closing the connection the 5 tuple { Protocol, Local IP, Local Port, Remote IP, Remote Port} goes into a TIME_WAIT state for 240 seconds by default. In this case, the protocol is fixed - TCP the local IP, remote IP and remote PORT are also typically fixed. So the variable is the local port. What happens is that when you don't bind, a port in the range 1024-5000 is used. So roughly you have 4000 ports. If you use all of them in 4 minutes - meaning roughly you make 16 web service calls per second for 4 minutes you will exhaust all the ports. That is the cause of this exception.

OK now how can this be fixed?

1. One of the ways is to increase the dynamic port range. The max by default is 5000. You can set this up to 65534. HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort is the key to use.

2. The second thing you can do is once the connection does get into an TIME_WAIT state you can reduce the time it is in that state, Default is 4 minutes, but you can set this to 30 seconds HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCPTimedWaitDelay is the key to use. Set this to 30 seconds

You're not closing the webrequest which might cause the connection to be open unecessarily long. This sounds like a perfect job for Parallel.Net's Parallel.Foreach, just be sure to indicate how many threads you want it running on

  ParallelOptions parallelOptions = new ParallelOptions();

        parallelOptions.MaxDegreeOfParallelism = 10;
        Parallel.ForEach(folderPathList, parallelOptions, folderPathList =>
        {
            using(WebRequest request = WebRequest.Create(folderPath))
            {
               request.Credentials = DefaultCredentials;
               request.Method = "MKCOL";

               GetResponse request = WebRequest.Create(folderPath);
               request.Credentials = DefaultCredentials;
               request.Method = "MKCOL";
               using (WebResponse response = request.GetResponse());
            }
        });

Another thing to keep in mind is maxConnections, be sure to set it in your app.config:

<configuration>
  <system.net>
    <connectionManagement>
      <add address = "*" maxconnection = "100" />
    </connectionManagement>
  </system.net>
</configuration>

Of couse in a real-world scenario you would have to add try-catch to and retrying connections that might time out leading to more complicated code

For this kind of IO intensive tasks, asynchronous programming model is very useful. However, it is a little hard to use in C#.C# also has language level support for async now, you can try the CTP release.