What are the risks of not patching a server or hypervisor for Meltdown?

Basically, if you run code from untrusted sources on a machine that has data you don't want that code to have access to, you need to patch. Desktop computers should be patched because they've got an unfortunate habit of encountering untrusted code; shared-hosting servers, particularly virtual private server hosts, must be patched, because Meltdown lets one user access every other user's data.

Note that the Meltdown attack cannot be used to break out of a virtual machine. You can break out of a container, sandbox, or a paravirtualized system, but performing the Meltdown attack in a fully-virtualized system just gets you access to that VM's kernel memory, not the host's kernel memory.

My understanding of the issue is that it is a local information leak, where local means that the information are leaked "only" to processes on the same physical hardware and not (directly) to remote systems. And, it is an attack which was shown to be actually usable in practice to extract sensitive information, even it is currently not trivially to exploit. But how easy the exploit is might quickly change as seen by Rowhammer, which evolved within a short time from only being a mostly theoretical problem to more reliable exploiting the issue using Javascript inside a browser or to root Android phones.

Thus, if there is the chance that some untrusted code is executed on the server you should patch. That's why all the larger cloud providers already patched their systems or will do shortly. And that's why the patches were so quickly incorporated into the Linux kernel, which is very unusual for changes to the memory subsystem.

Note that untrusted code might not only be run if you have untrusted users on the system. It can also happen if you process data originating from an untrusted source. For example an attacker could use existing functionality of your web server to upload an image which then gets converted on your server (i.e. scaling or similar). Given the history of bugs in graphic libraries it would not be unlikely that this conversation could result in code execution. And giving the nature of the issue I doubt that sandboxes, docker or similar will stop exploitation of the bug.