ulimit not limiting memory usage

Your example should work like you think (program gets killed after consuming too much RAM). I just did a small test on my shell server:

First I restricted my limits to be REALLY low:

ulimit -m 10
ulimit -v 10

That lead to about everything getting killed. ls, date and other small commands will be shot before they even begin.

What Linux distribution you use? Does your program use only a single process or does it spawn tons of child processes? In the latter case ulimit might not always be effective.

ulimit -m no longer works. Use ulimit -v instead.

The reason is that ulimit calls setrlimit, and man setrlimit says:

RLIMIT_RSS Specifies the limit (in bytes) of the process's resident set (the number of virtual pages resident in RAM). This limit has effect only in Linux 2.4.x, x < 30, and there affects only calls to madvise(2) specifying MADV_WILLNEED.