Prevent a script exhausing system resources and crashing entire system

Alternative #1: Monitor your process with monit

Install M/Monit and create a configuration file based on this template:

check process myprogram
matching "myprogram.*"
start program = "/usr/bin/myprogram" with timeout 10 seconds
stop program = "/usr/bin/pkill thatscript"
if cpu > 99% for 2 cycles then stop
if loadavg (5min) > 80 for 10 cycles then stop

Alternative #2: Limit process CPU usage with cgroups

The most native Linux specific solution of them all. Offers a lot of options and complexity.

Example:

sudo cgcreate -g cpu:/cpulimited
sudo cgset -r cpu.shares=512 cpulimited
sudo cgexec -g cpu:cpulimited /usr/bin/myprogram > /dev/null &

I encourage you to read more at:

DigitalOcean - How-to: Limit resources using cgroups on CentOS 6

RedHat - Resource Management Guide

Oracle - List of cgroups subsystems

Alternative #3: Limit process CPU usage with cpulimit

Get latest version of cpulimit from your package manager of choice, or by getting the source available at GitHub.

Limit the CPU usage to 90%: cpulimit -l 90 /usr/bin/myprogram > /dev/null &

Sidenote:

You can also pin a certain process to use certain CPU core(s) to ensure that you always have some free CPU power.

systemd can limit a target/service's resources. From the man page:

CPUQuota:

Assign the specified CPU time quota to the processes executed. Takes a percentage value, suffixed with "%". The percentage specifies how much CPU time the unit shall get at maximum, relative to the total CPU time available on one CPU. Use values > 100% for allotting CPU time on more than one CPU. This controls the "cpu.max" attribute on the unified control group hierarchy and "cpu.cfs_quota_us" on legacy. For details about these control group attributes, see cgroup-v2.txt and sched-design-CFS.txt.

Example: CPUQuota=20% ensures that the executed processes will never get more than 20% CPU time on one CPU.

Implies "CPUAccounting=true".

Since using this implies CPUAccounting I'll include that as well

CPUAccounting:

Turn on CPU usage accounting for this unit. Takes a boolean argument. Note that turning on CPU accounting for one unit will also implicitly turn it on for all units contained in the same slice and for all its parent slices and the units contained therein. The system default for this setting may be controlled with DefaultCPUAccounting= in systemd-system.conf(5).

I'll also quote from Slice:

The name of the slice unit to place the unit in. Defaults to system.slice for all non-instantiated units of all unit types (except for slice units themselves see below). Instance units are by default placed in a subslice of system.slice that is named after the template name.

So by default everything will get thrown into the same slice, which means everything in a single resource pool.

There's also MemoryHigh to look at:

MemoryHigh:

Specify the high limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.

Takes a memory size in bytes. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system. If assigned the special value "infinity", no memory limit is applied. This controls the "memory.high" control group attribute. For details about this control group attribute, see cgroup-v2.txt.

Implies "MemoryAccounting=true".

This setting is supported only if the unified control group hierarchy is used and disables MemoryLimit=.

---

You can easily toss a script into a systemd service.

Assuming /usr/local/thatscript.sh is the script:

/usr/lib/systemd/system/thatscript.service

[Unit]
Description=This runs "thatscript"
ConditionFileNotEmpty=/usr/local/thatscript.sh

[Service]
Type=simple
ExecStart=/usr/local/thatscript.sh
CPUQuota=20%

[Install]
WantedBy=multi-user.target

Then you'll need systemctl daemon-reload to read in the new service file, then you can systemctl enable thatscript.service if you want it to run at boot, or systemctl start thatscript.service if you want to start it manually.