2008 R2 Terminal Server: "Insufficient system resources exist to complete the requested service"
Solution 1:
This has been solved.
I began to examine the registry because increasing CPU and RAM resources on the virtual machine did not resolve the issue.
I was pointed to Microsoft's dureg tool to estimate the registry's size. Browsing via regedit, I encountered issues opening the keys under HKEY_USERS\.Default\PRINTERS. Using dureg, I started probing under that hierarchy.
Printers were the problem. The cause and fix are detailed in:
The size of the "HKEY_USERS.DEFAULT" registry hive continuously increases on a Windows Server 2008 R2 SP1-based server
Hotfix: http://support.microsoft.com/kb/2871131
This apparently stops the growth, but the keys and registry need to be compressed to reclaim space.
Compressing bloated registry: http://support.microsoft.com/kb/2498915
1) Boot from a WinPE disk.
2) Open regedit while booted in WinPe, load the bloated hive under HLKM. (e.g. HKLM\Bloated)
3) Once the bloated hive has been loaded, export the loaded hive as a "Registry Hive" file with a unique name.
4) Unload the bloated hive from regedit.
5) Rename the hives so that you will boot with the compressed hive.
e.g.
c:\windows\system32\config\ren software software.old
c:\windows\system32\config\ren compressedhive software
Hmm, a few steps... kinda tricky to do remotely during production hours. I tried to reach out to my resident Microsoft expert to complete, but he was busy chasing down some SCCM or SCVMM issue somewhere. Reading through some Citrix-related forums, I took note of a tool that could perform the above with fewer steps...
So I took a virtual machine snapshot, then downloaded and ran freeware registry compression software (Tweaking.com); despite the overwhelming sound of the collective groans of Microsoft systems engineers everywhere...
note the 1.4GB saved in the default Config...
PLEASE REBOOT!
Following a reboot, all was well. The user count reached 86 with no ill effects and no profile-related errors. I've monitored the printer registry hive and it's held stable.
Solution 2:
In Windows Server 2003 that error was a result of kernel memory exhaustion. Because you're dealing with Windows Server 2008 R2 I'm not sure how closely related the cause of the problem is to the cause in W2K3, but I would bet that it is a memory issue due to the number of users and processes. I would take a look at Nonpaged Pool memory exhaustion as the probable cause. In addition, the number of procceses is at almost 800, which is quite high. MS would probably tell you to reduce the number of processes, which can only be done by reducing the user load.
This article has some good information regarding memory usage in Windows and how you can view the Nonpaged Pool limit to see if that's the cause of the problem:
https://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx
Solution 3:
Start up Windows Performance Monitor to monitor the various counters:
- Context Switches
- Page Table Entries
- GDI elements
- Handles
- … (whatever you can find)
And see if one of these peaks when you get a failed login.
Also: something is causing high kernel CPU% on your system - you should investigate that to see if it leads you to a related problem.
The User Profile Hive Cleanup service may help out here as it "helps to ensure user sessions are completely terminated when a user logs off".