Create child processes inside a child process with Python multiprocessing failed
As noxdafox said, multiprocessing.Pool
uses daemonic processes. I found a simple workaround that uses multiprocess.Process
instead:
Parent program:
import multiprocessing
import child_process
processes = [None] * 4
for i in range(4):
processes[i] = multiprocessing.Process(target=child_process.run, args=(i,))
processes[i].start()
for i in range(4):
processes[i].join()
Child program (with name child_process.py
):
import multiprocessing
def test(info):
print 'TEST', info[0], info[1]
def run(proc_id):
pool = multiprocessing.Pool(processes=4)
pool.map(test, [(proc_id, i) for i in range(4)])
pool.close()
pool.join()
The output is 16 lines of TEST
:
TEST 0 0
TEST 0 1
TEST 0 3
TEST 0 2
TEST 2 0
TEST 2 1
TEST 2 2
TEST 2 3
TEST 3 0
TEST 3 1
TEST 3 3
TEST 3 2
TEST 1 0
TEST 1 1
TEST 1 2
TEST 1 3
According to the documentation for multiprocessing, daemonic processes cannot spawn child processes.
multiprocessing.Pool
uses daemonic processes to ensure that they don't leak when your program exits.