Why does multiprocessing Julia break my module imports?

First it is important to note that you are NOT using multi-thread parallelism, you are using distributed parallelism. When you initiate with -p 2 you are launching two different processes that do not share the same memory. Additionally, the project is only being loaded in the master process, that is why the other processes cannot see whatever is in the project. You can learn more about the different kinds of parallelism that Julia offers in the official documentation.

To load the environment in all the workers, you can add this to the beginning of your file.

using Distributed
addprocs(2; exeflags="--project")
@everywhere using Solver
@everywhere include("src/main.jl")

and remove the -p 2 part of the line which you launch julia with. This will load the project on all the processes. The @everywhere macro is used to indicate all the process to perform the given task. This part of the docs explains it.

Be aware, however, that parallelism doesn't work automatically, so if your software is not written with distributed parallelism in mind, it may not get any benefit from the newly launched workers.


There is an issue with Julia when an uncompiled module exists and several parallel processes try to compile it at the same time for the first use.

Hence, if you are running your own module across many processes on a single machine you always need to run in the following way (this assumes that Julia process is run in the same folder where your project is located):

using Distributed, Pkg
@everywhere using Distributed, Pkg
Pkg.activate(".")
@everywhere Pkg.activate(".")
using YourModuleName
@everywhere using YourModuleName

I think this approach is undocumented but I found it experimentally to be most robust. If you do not use my pattern sometimes (not always!) a compiler chase occurs and strange things tend to happen.

Note that if you are running a distributed cluster you need to modify the code above to run the initialization on a single worker from each node and than on all workers.