Can functions know if they are already multiprocessed in Python (joblib)

The answer to the specific question is: I don't know of a ready-made utility.

A minimal(*) core refactoring would to be add a named parameter to your function currently creating child processes. The default parameter would be your current behavior, and an other value would switch to a behavior compatible with how you are running tests(**).

(*: there might be other, may be better, design alternatives to consider but we do not have enough information) (**: one may say that the introduction of a conditional behavior would require to test that as well, and we are back to square one...)


Parallel in joblib should be able to sort these things out:
http://pydoc.net/Python/joblib/0.8.3-r1/joblib.parallel/

Two pieces from 0.8.3-r1:

# Set an environment variable to avoid infinite loops
os.environ[JOBLIB_SPAWNED_PROCESS] = '1'

Don't know why they go from a variable referring to the environmental, to the env. itself.. But as you can see. The feature is already implemented in joblib.

# We can now allow subprocesses again
os.environ.pop('__JOBLIB_SPAWNED_PARALLEL__', 0)


Here you can select other versions, if that's more relevant:
http://pydoc.net/Python/joblib/0.8.3-r1/