How to solve this memory issue gracefully?

Someone suggested in your hear cgroups. Well, try to seek that direction as it can provide you with:

• applied to a group of task you choose (thus not system wide but neither per process)
• the limits are set for the group
• the limits are static
• they can enforce hard limit on memory and/or memory+swap

Something like that could bring you closer to your goals:

group limited {
  memory {
    memory.limit_in_bytes = 50M;
    memory.memsw.limit_in_bytes = 50M;
  }
}

This tells that the tasks under this cgroup can use at maximum 50M of memory only and 50M of memory+swap, so when the memory is full, it won't swap, but if the memory is not full and some data could be mapped in swap, this could be allowed.

Here is an excerpt from the cgroup's memory documentation:

By using memsw limit, you can avoid system OOM which can be caused by swap shortage.