Is it true that in multiprocessing, each process gets it's own GIL in CPython? How different is that from creating new runtimes?
The short answer to the first title question is: Yes. Each process has its own Global Interpret Lock. After that, it gets complicated and not really as much a Python matter as it is a question for your underlying OS.
On Linux, it should be cheaper to spawn new processes through multiprocessing
rather than starting a new Python interpreter (from scratch):
- you
fork()
the parent process (side note:clone()
is actually used these days), the child already has your code and starts with a copy of parents address space -> since you are actually spawning another instance of your running process no need toexecve()
(and all the overhead associated with that) and repopulate its content. - actually when we say copy of the address space, it actually does not get all copied, but would rather use copy-on-write; so unless you modified it, you do not need to copy it at all.
For that reason my gut feeling would be, multiprocessing would almost always be more efficient than starting a completely new Python interpreter from scratch. After all, even you started a new interpreter (presumably from the running process), it first performs the fork()
/clone()
including "copy" of the parent's address space before moving onto execve()
.
But really this may vary and depends on how your underlying OS handles creation of new processes as well as its memory management.