What is the purpose of the Visual Studio Hosting Process?

From MSDN:

The Visual Studio hosting process improves debugger performance and enables new debugger features, such as partial-trust debugging and design-time expression evaluation

The MSDN library doesn't give very good info on the "hosting process". The last two features listed in Eric's link are actually problems induced by the feature. There's another one that you're bound to run into sooner or later: it uses a different app.config file. The active one is named yourapp.vshost.exe.config. Watch out for this when you make manual changes to the file.

Another feature it supports that's very visible when you debug your app but isn't mentioned anywhere is what happens to the output produced by Console.Write(). In a non-console mode app, it gets redirected to the IDE's Output window. Very useful.

The term "hosting" refers to a feature of the CLR, it can be "hosted". Examples of custom CLR hosts are SQL Server and ASP.NET. Hosting allows one to configure the CLR before it gets started. One primary use of this is configuring the primary AppDomain and setting up custom security policies. Which is exactly what the hosting process is doing.

A good example of a custom CLR host is available in this question.

Long story short: in debug mode you are running with a customized version of the CLR, one that improves the debugging experience.