How do I use oom_score_adj?

^{Based on my own Q&A on Unix&Linux on a similar question.}

As Stuart pointed out very well in his answer, valid values are integers in the range of -1000 to 1000 for oom_score_adj. The lower the value, the lower the chance that it's going to be killed.

It's very inconvenient to have to change this value over and over again once you restart the application. The information is simply lost after the process has terminated. Upstart (the init daemon in Ubuntu), has a nice option for this to configure for daemons to make sure it's set whenever it has (re)started:

oom score

[...] snip [...]

Example:

# this application is a "resource hog"
oom score 1000

expect daemon
respawn
exec /usr/bin/leaky-app

So, basically, you can edit the /etc/init/myservice.conf configuration file for the services you like to change, to include a line oom score -1000. I do assume that the services 'pbs' or 'gridengine' you're talking about in your question is Upstart-enabled, otherwise you'll need another way to change this permanently.

If you can modify the start script spawning the process that you want to prioritize for oom-killer, add the following to the start script:

echo 1000 > /proc/self/oom_score_adj

The value is inherited for child processes.

If you set a high(ish) value for either oom_adj or oom_score_adj then they will be killed first. e.g.

echo 15 > /proc/[pid]/oom_adj

oom_adj goes from -16 to 15 and as mentioned above oom_score_adj accepts -1000 to 1000