OS with encrypted RAM?

I also would like to know if it would be theoretically possible (whilst obviously difficult) to translate an existing JIT compiler (e.g. .NET CLR) that produces code that automatically encrypts its user-mode memory.

This might be quite difficult for the OS to do on behalf of a program. The reason for this is that the virtual pages allocated by the operating system are in fact controlled by the CPU - the OS instructs the CPU as to what it wants in terms of storage areas and the CPU then computes the offsets on that basis for each program and applies the result directly. Have a read up on the MMU and Paging on x86.

In short, when your program makes the computation mov eax, [edi+something], the MMU/paging handles the address translation and issues a page fault when the page is not found. That allows you to then load the page out of storage, should you need to.

So the memory access doesn't go through the kernel per se and as such can't be hooked and processed as a file read or write can (your read and writes are translated through the appropriate system calls into the appropriate file system structure. As such, the OS sees the data as it passes through. You don't need a system call to write to RAM. You could make one, but it would only work for programs that call it and so not most programs).

However, that doesn't mean to say a JIT process couldn't do this on behalf of an application, or the application itself couldn't keep data encrypted until it needs to load it into registers, decrypting it as it passes over the data.

In that case, you're into the issue SteveS covers well - you have a key storage problem. The key itself must also be in RAM somewhere. This is a turtles all the way down problem - fundamentally it's impossible to keep that key "secure".

Finally, since being able to read another application's RAM requires supervisor mode access to the CPU (i.e. kernel space) you probably have bigger issues if your concern is software interception. If your concern is hardware, I think physical security might be a better way of mitigating the risk.

Edit I had a look for papers. Here's one called CryptKeeper. Their technique is to have a large encrypted "RAM disk" as the swap file and swap all their pages out to that when not used:

We mitigate this vulnerability with Cryptkeeper (CK), a software-encrypted virtual memory manager. Traditional pro- cessors cannot operate on encrypted data, so CK segments RAM into a smaller working set called the Clear, and a larger encrypted RAM device called the Crypt. As the working space fills, pages are automatically swapped into the encrypted portion of memory, and are decrypted on demand.

Apparently the performance isn't too bad, but I don't think there are any implementations of it yet.

Edit 2 So it turns out CryptKeeper and the OpenBSD Swap Encryption mechanisms work on a very similar level; it does not actually encrypt physical memory but uses physical memory as a swap structure, forcing page faults and encrypting/decrypting data in resolution.

EDIT from question author, December 2018: AMD now supports the SME instruction set extension which allows for hardware encryption of RAM pages, plus TSME for full-memory encryption, and SEV for use with encrypted memory in virtualisation. This appears to enable full-system memory encryption on modern AMD64 platforms.

The OpenBSD operating system includes automatic encryption of virtual memory; it is enabled by default since version 3.9.

Without a CPU with embedded encryption hardware, the data in caches and physical RAM cannot be really encrypted, because the CPU could then not use it; but "virtual memory" as in "RAM blocks copied to and from the disk" is under the total control of the operating system, and thus the OS can encrypt it at will, which is what OpenBSD does. To some extent, most of the RAM could be kept encrypted as well, by using it as if it was a disk (i.e. a swap file on a RAM-based virtual disk -- it would work !).

RAM contents are supposed to disappear automatically when power is cut (it actually takes about one minute or so, depending on temperature).

I'd be curious if something exists too, but I'm not entirely sure one can exist without hardware intervention.

My logic is that in order for the memory to be encrypted, the key has to be known by whatever process is accessing it directly, which for the sake of argument would just be at the kernel level. At some point that key needs to be moved into unencrypted memory so things can be encrypted or decrypted, since the processor can't manipulate encrypted data directly. Once this key is in unencrypted memory you can access it through a couple ways: memory dump, kernel driver, physical probing, etc.

In theory you could prevent memory dumps from being written to disk and could prevent drivers from being installed, and you could lock the device in a safe and drop it into the ocean (a waterproof safe preferably) making it physically inaccessible, but that changes the usage of the system dramatically.

Well, let me rephrase my first statement then -- one could exist without hardware intervention, but I'm not sure how secure it would be.

As for your other question, it would be possible, but you still run into the same basic problem. You would have to modify the compiler and modify the JIT interpreter, and the exe would store the key in its own memory space. The exe would have to hand the key off to the interpreter to handle the crypto. The problem with this is that you still have to store the key somewhere in unencrypted memory. In fact, if the key is hard coded in the exe, then you could just crack open the file and go looking for it.

Nevertheless, it would be something cool to look at.