Optimizing multiprocessing.Pool with expensive initialization
most obvious, lazy load
_foo = None
def f(y):
global _foo
if not _foo:
_foo = Foo()
return _foo.run(y)
The intended way to deal with things like this is via the optional initializer
and initargs
arguments to the Pool()
constructor. They exist precisely to give you a way to do stuff exactly once when a worker process is created. So, e.g., add:
def init():
global foo
foo = Foo()
and change the Pool
creation to:
pool = mp.Pool(4, initializer=init)
If you needed to pass arguments to your per-process initialization function, then you'd also add an appropriate initargs=...
argument.
Note: of course you should also remove the
foo = Foo()
line from f()
, so that your function uses the global foo
created by init()
.