How to limit memory usage by application in linux?

Solution 1:

'ulimit' is a 'per-application' control… per process in fact. The ulimit shell command is a shell built-in setting the limit for the shell process and its children. Put the 'ulimit' command in the script starting your application and the limit will be set for this application only.

Solution 2:

On systemd-based distros you can also use systemd-run (which indirectly uses cgroups). For example:

systemd-run --scope -p MemoryLimit=1G firefox

Note: this gonna ask you for a password but the app gets launched as your user. Do not allow this to mislead you into thinking that the command needs to run with sudo, because that would cause the command to run under root, which hardly was your intention.

If you want to not enter the password (indeed, why would you need a password to limit memory you already own), you could use --user option, however for this to work you gonna need cgroupsv2 support enabled, which right now requires to boot with systemd.unified_cgroup_hierarchy kernel parameter.