How can we troubleshoot intermittent "An existing connection was forcibly closed" errors caused by a Cisco CSS

The problem was the Cisco CSS. We determined this by pointing the tier 1 servers directly to the tier 2 servers and going 48 hours without observing the problem. Once we determined it was the CSS, we corrected this problem by adjusting the insanely low default value for this parameter:

"Default flow inactivity timeouts, in seconds, for the TCP or UDP port. If a flow is idle for the amount of time specified in the timeout value, the CSS tears down the flow and reclaims the flow resources."

We set this to 84 (which is 84 16-second increments). Since the default keep-alive for HTTP is 120 seconds, the default value was too low.

To check recycling of the Application pool go to your IIS and open the Properties of the Application Pool on which your remoting service is running. You can configure recycling of Application pools using a time interval, number of requests or define specific times.

You could remove the current recycling rules and set a recycling to a time where no connections are expected, like 3.00 at night. Then see if the exceptions stil occur.

It could be a network component causing this. The way to rule this out would be to place both machines (or test machines) on the same subnet, then run a load test, and verify that you do not get the same error.

The other things that could be causing it could be:

• A timeout, try increasing the timeout values
• Too large message size, try increasing the size of the message allowed, also size of request allowed in IIS
• You could be hitting some max value, such as max calls or max connections see: http://msdn.microsoft.com/en-us/library/ee377061(v=bts.10).aspx