Implement Parallel for loops in Python
You can also use concurrent.futures
in Python 3, which is a simpler interface than multiprocessing
. See this for more details about differences.
from concurrent import futures
total_error = 0
with futures.ProcessPoolExecutor() as pool:
for error in pool.map(some_function_call, parameters1, parameters2):
total_error += error
In this case, parameters1
and parameters2
should be a list or iterable of the same size as the number of times you want to run the function (24 times as per your example).
If paramters<1,2>
are not iterables/mappable, but you just want to run the function 24 times, you can submit the jobs for the function for the required number of times, and later acquire the result using a callback.
class TotalError:
def __init__(self):
self.value = 0
def __call__(self, r):
self.value += r.result()
total_error = TotalError()
with futures.ProcessPoolExecutor() as pool:
for i in range(24):
future_result = pool.submit(some_function_call, parameters1, parameters2)
future_result.add_done_callback(total_error)
print(total_error.value)
You can use python multiprocessing
:
from multiprocessing import Pool, freeze_support, cpu_count
import os
all_args = [(parameters1, parameters2) for i in range(24)]
# call freeze_support() if in Windows
if os.name == "nt":
freeze_support()
# you can use whatever, but your machine core count is usually a good choice (although maybe not the best)
pool = Pool(cpu_count())
def wrapped_some_function_call(args):
"""
we need to wrap the call to unpack the parameters
we build before as a tuple for being able to use pool.map
"""
sume_function_call(*args)
results = pool.map(wrapped_some_function_call, all_args)
total_error = sum(results)