Does a Virtual Machine stop malware from doing harm?
Theoretically, the guest system is totally isolated by the VM and cannot even "see" the host, let alone attack it; so the guest cannot break out of the VM. Of course, in practice, it has occasionally happened (web archive link). An attack requires exploiting a security issue (i.e. a programming bug which turns out to have nasty consequences) in the VM implementation or, possibly, the hardware features on which the VM builds on. There are few exit routes for data out of the VM; e.g., for Internet access, the VM is emulating a virtual network card, which deals only with the lowest level packets, not full TCP/IP -- thus, most IP-stack issues remain confined within the VM itself. So bugs leading to breakout from VM tend to remain rare occurrences.
There are some kinds of attacks against which VM are very effective, e.g. fork bombs. From the point of view of the host system, the VM is a single process. A fork bomb in the guest will bring to its knees the scheduler in the guest OS, but for the host this will be totally harmless. Similarly for memory: the VM emulates a physical machine with a given amount of RAM, and will need about that amount of "real" RAM to back it up efficiently. Regardless of what the guest does, the VM will never monopolize more RAM than that. (You still want to limit VM RAM size to, say, at most 1/2 of your physical RAM size, because the extra "real" RAM is handy for disk caching; and the host OS will want to use some, too.)
Disclaimer: I'm going for a relatively high-level understanding. If you want a detailed guide, that's out of scope. Additionally, there are other ways (entirely in software) to implement virtual machines to which this does not apply. I am also focusing on "breaking out" through the virtualisation mechanisms only - i.e. not ones that can happen PC-to-PC on actual hard networked hosts.
I like detail, so here we go with some. Firstly, codeproject has some excellent assembler references on the different modes of an x86 CPU (real, protected and long) and use of virtualisation. There's an Intel VT blog (I'm not sure if Intel write this) and lastly the first part of the Rootkit Arsenal is dedicated to explaining x86 and is an excellent read, complete with walkthroughs and nice diagrams. Understanding it all takes patience, so what I'll do here is to give a very brief introduction to how it works.
The way we rocked when we ran DOS
DOS and early 16-bit real mode systems operate a segmented memory model. There is no control over the size of segments and there are no protection switches on any of those segments. Code gets loaded into a segment of memory and it runs; it can far jump into other segments, so any code, anywhere can alter anything, including producing a TSR (terminate and stay resident) piece of code that simply points one of the IVT entries (interrupt vector table) at an address in its space, before executing the original. Basically, there is no protection. None. Nada.
The rise of 32-bit protected mode
Protected mode gets complicated quickly. There are three parts to it - segmentation, paging and PAE. Each requires a table of data that tells the CPU about that segment, page or helps it extend the address space (PAE). These include the famous ring flags (they apply to segments and pages) which implement process isolation. Paging is your way to load data out of RAM and onto disk and create fancy things like virtual memory (see, the word virtual! We're getting there!)
Long mode
Long mode does away with segmentation and simply mandates the PAE/Paging structures. Again, to totally trivialise implementing an OS, Paging is controlled by structures in memory which are then set up via special instructions. Voila, one can achieve process isolation with the right settings. Again, I'm trivialising slightly...
Give me virtualisation!
Okay. Virtualisation is the same general concept. Virtual machines are set up using virtual machine control structures which dictate how their memory is mapped back to physical memory, kinda like paging. Crucially, under certain conditions the virtual machine will be required to ask the host operating system for something, kinda like process isolation, kinda like a software interrupt. These are referred to VM exits and provide information to the host such as the state of registers on exit. Kinda just like a system call.
Can a piece of malware break out of a virtual machine?
So, as far as the VM is concerned, the host OS has all its own memory space and can be infected/damaged/destroyed as it pleases.
In terms of affecting the host memory directly, the virtual machine cannot, because it cannot see it. The host must map the required memory into the virtual machine space. It must also, in that memory space, implement everything from the BIOS up. In order to communicate with certain host devices for certain tasks, the host machine must set up those VM exit conditions and the target VM must trigger them. When that happens, control is transferred to the host.
There are, therefore, two possible at-risk areas:
- The actions the host takes in response to a VM exit. If there are any bugs in this handling it may be possible to persuade the host to execute something it shouldn't.
- Any host access to the guest machine's memory space. Remember the host machine code running in ring 0 can waltz in and crash the party wherever it so pleases. It just so happens that you can set the guest's memory from the guest (surprisingly).
This leads you to your exploit mechanism. You need a handling bug in the VM exit routine, then you need to be able to persuade that code to execute some memory, ideally code you just put into a page from the guest vm. Once done, say bye to Kansas.
As Tom Leek says, VMs are incredibly effective in defending against fork bombs. In a similar vein to the way the OS can limit how much memory a process can allocate, so it can limit how much memory is mapped to the VM. Run out and the guest OS believes it is out of physical memory; the host will not allocate it more unless you implement a VM exit to do this, which would be a bit dangerous and I do not believe this is done.
How likely is this?
Not very. It depends on those VM exit implementations entirely, or reading memory from the guest on the host with a nice bug in your reading code. It also requires that said bug would allow you to control the crash in such a way that you can force execution to the memory address your host holds. The VM exit must be able to access that memory.
What I have not covered?
- Attacks on existing software stacks such as TCPIP. The vulnerabilities here are the same as if you had two actual phsyical PCs anyway.
- Entirely software implemented virtualisation.
- Virtualisation on any other type of chip. This applies to Intel VT compatible setups.
Finally, I have previously argued that process isolation is a form of sandboxing. Reading that answer and this one, you should now be able to understand why I define them that way. There are remarkable similarities between process isolation and virtual machines in x86.
Update
So, I've dug around even more on this, especially into the blue pill research. What I've described is a very simplistic high level view. I've found more detail. Here's a whole paper dedicated to it from Invisible Things Lab. It turns out that their defences talk included the concept of denying execute access to user mode pages from ring 0, thereby preventing direct execution of data the virtual machine has placed into memory. It turns out this is being implemented in Intel CPUs and patches currently exist in the Linux Kernel. So, depending on how that goes, it may well be the case that attacks of this nature become much harder, even if exploits exist.
I have done quite a bit of malware experimentation within a VM - mostly using backtrack4 to break in from one host to the next. I'm a VMware Workstation user, primarily.
The largest issue comes from the possibility of the network connection of your VM transferring back to your host OS. You want to totally disable networking and / or use a network which has no access back to your host.
Limiting memory is a good best practice. I generally keep it to about a quarter, same as you. The CPU time is limited to either the number of cores or (if you have finer grained controls in your software) the percentage of CPU time defined for your specific VM.
Targeted attacks capable of breaking out of the virtual environment do exist and are commercially available - such as the cloudburst @Hendrick mentions - but are relatively rare. Keeping up to date on your virtualization patches is a very good idea.
See here, here, and here for details.