Limiting process memory/CPU usage on linux

You might want to investigate cgroups as well as the older (obsolete) ulimit.

In theory, you should be able to use ulimit for this purpose. However, I've personally never gotten it to work.