Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python
The previously accepted solution has race conditions and it does not work with map
and async
functions.
The correct way to handle Ctrl+C/SIGINT
with multiprocessing.Pool
is to:
- Make the process ignore
SIGINT
before a processPool
is created. This way created child processes inheritSIGINT
handler. - Restore the original
SIGINT
handler in the parent process after aPool
has been created. - Use
map_async
andapply_async
instead of blockingmap
andapply
. - Wait on the results with timeout because the default blocking waits to ignore all signals. This is Python bug https://bugs.python.org/issue8296.
Putting it together:
#!/bin/env python
from __future__ import print_function
import multiprocessing
import os
import signal
import time
def run_worker(delay):
print("In a worker process", os.getpid())
time.sleep(delay)
def main():
print("Initializng 2 workers")
original_sigint_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
pool = multiprocessing.Pool(2)
signal.signal(signal.SIGINT, original_sigint_handler)
try:
print("Starting 2 jobs of 5 seconds each")
res = pool.map_async(run_worker, [5, 5])
print("Waiting for results")
res.get(60) # Without the timeout this blocking call ignores all signals.
except KeyboardInterrupt:
print("Caught KeyboardInterrupt, terminating workers")
pool.terminate()
else:
print("Normal termination")
pool.close()
pool.join()
if __name__ == "__main__":
main()
As @YakovShklarov noted, there is a window of time between ignoring the signal and unignoring it in the parent process, during which the signal can be lost. Using pthread_sigmask
instead to temporarily block the delivery of the signal in the parent process would prevent the signal from being lost, however, it is not available in Python-2.
The solution is based on this link and this link and it solved the problem, I had to moved to Pool
though:
import multiprocessing
import time
import signal
import sys
def init_worker():
signal.signal(signal.SIGINT, signal.SIG_IGN)
def worker():
while(True):
time.sleep(1.1234)
print "Working..."
if __name__ == "__main__":
pool = multiprocessing.Pool(50, init_worker)
try:
for i in range(50):
pool.apply_async(worker)
time.sleep(10)
pool.close()
pool.join()
except KeyboardInterrupt:
print "Caught KeyboardInterrupt, terminating workers"
pool.terminate()
pool.join()
Just handle KeyboardInterrupt-SystemExit exceptions in your worker process:
def worker():
while(True):
try:
msg = self.msg_queue.get()
except (KeyboardInterrupt, SystemExit):
print("Exiting...")
break