setting ulimit on a running process

On recent versions of Linux (since 2.6.36), you can use the prlimit command and system call to set resource limits on an arbitrary process (given appropriate permissions):

$ prlimit --core=unlimited: --pid $$
$ prlimit --core --pid $$
RESOURCE DESCRIPTION             SOFT      HARD UNITS
CORE     max core file size unlimited unlimited blocks

You need util-linux-2.21 for the prlimit command, but you should be able to throw together a quick program to invoke the prlimit system call otherwise:

int prlimit(pid_t pid, int resource, const struct rlimit *new_limit, struct rlimit *old_limit);

If you don't have a new enough version of Linux (or another OS) the only fix I'm aware of is to connect to the process with gdb and issue setrlimit from the debugger:

$ gdb -p $PID
...
(gdb) set $rlim = &{0ll, 0ll}
(gdb) print getrlimit(9, $rlim)
$1 = 0
(gdb) print *$rlim
$2 = {-1, -1}
(gdb) set *$rlim[0] = 1024*1024
(gdb) print setrlimit(9, $rlim)
$3 = 0

This is for setting ulimit -m, RLIMIT_AS = 9; exactly the same applies for ulimit -c (RLIMIT_CORE, numeric value 4 on Linux on x86-64). For "unlimited", use RLIM_INFINITY, usually -1. You should check in /usr/include/bits/types.h what the size of rlim_t is; I'm assuming long long (it's actually unsigned, but using a signed type makes "unlimited" -1 easier to read).

As Ubuntu 14.04 Trusty has no util-linux-2.21 (it is 2.20), there is no prlimit CLI command to use.

Using Python3.4+ (which is available on Ubuntu 14.04 and all later versions) can set resource limit for a running process. Run as root:

1-liner:

# PID=966
# grep 'open file' /proc/$PID/limits
Max open files            1024                 4096                 files     
# python3 -c "import resource; resource.prlimit($PID, resource.RLIMIT_NOFILE, (2048, 12345))"
# grep 'open file' /proc/$PID/limits
Max open files            2048                 12345                files   

Or more verbose:

# python3
Python 3.4.3 (default, Nov 28 2017, 16:41:13) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
>>> import os
>>> resource.prlimit(os.getpid(), resource.RLIMIT_NOFILE)
(1024, 4096)
>>> resource.prlimit(os.getpid(), resource.RLIMIT_NOFILE, (1369, 9999))
(1024, 4096)
>>> resource.prlimit(os.getpid(), resource.RLIMIT_NOFILE)
(1369, 9999)

Verify it works:

# grep 'open file' /proc/1472/limits 
Max open files            1369                 9999                 files 

Note this works with Linux 2.6.36 or later with glibc 2.13 or later.