Memory limiting solutions for greedy applications that can crash OS?

The shell built-in ulimit allows you to restrict resources. For your case, to limit memory use in the shell (and its children), use ulimit -v.

Demonstration setting a memory limit of 100 MB (100000 KB):

$ ulimit -v
unlimited
$ python -c '[ "x" * 100000000 ]'
$ ulimit -v 100000
$ python -c '[ "x" * 100000000 ]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
MemoryError

It's observed using ps uww -C script-name-here that python requires at least 29MB of memory (VSZ column). The RSS limit grows as your python script needs more memory so adapt that column.

Cgroups should let you limit your memory usage on a per process basis.

https://en.wikipedia.org/wiki/Cgroups

http://www.mjmwired.net/kernel/Documentation/cgroups/memory.txt

Scientific computing is notoriously memory intensive, by sandboxing your app in a cgroup, the rest of the processes should not become victims as memory pressure will be alleviated.

Alternatively, a VM could be used as a sort of hard limit as the app can only use the memory delegated to the virtual machine, at the expense of performance of course. However a VM is much easier to configure for the uninitiated when compared to setting up and maintaining a cgroup.

Decisions decisions :) Good luck!