What is a virtualized process?

It's basically a file system and registry 'wrapper' that redirects file write attempts if the user does not have the correct write permissions, see the Wikipedia article about UAC for more information:

Applications written with the assumption that the user will be running with administrator privileges experienced problems in earlier versions of Windows when run from limited user accounts, often because they attempted to write to machine-wide or system directories (such as Program Files) or registry keys (notably HKLM). UAC attempts to alleviate this using File and Registry Virtualization, which redirects writes (and subsequent reads) to a per-user location within the user's profile.

For example, if an application attempts to write to a directory such as "C:\Program Files\appname\settings.ini" to which the user does not have write permission, the write will be redirected to "C:\Users\username\AppData\Local\VirtualStore\Program Files\appname\settings.ini". The redirection feature is only provided for non-elevated 32-bit applications, and only if they do not include a manifest that requests specific privileges.

Journeyman Geek explains what is virtualization. I'll explain how Windows determine need of virtualization.

OS looks in application manifest file (or PE-resource) and if manifest is not found at all or does not have proper compatibility section - Windows assumes that application is "old" and enables virtualization.

<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
 <application>
   <supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"></supportedOS>
   <supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"></supportedOS>
 </application>
</compatibility>

PS. Virtualization works for registry (sub)keys like HKEY_LOCAL_MACHINE to HKEY_CURRENT_USER\Software\Classes\VirtualStore\MACHINE.