Using multiprocessing pool in Python
Edit: You made an edit to your code so now my answer below is out of date. The only two things I think need doing are:
- add an
error_callback
because I still think you need to ensure that the pool as written does not fail silently by default. - rewrite
multiprocessing.current_process().name()
asmultiprocessing.current_process().name
.
So:
import multiprocessing
results = []
def log_results(result):
results.append(result)
def log_e(e):
print(e)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name}")
return x * y
pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
print (f"Checking x {x} and y {y}")
pool.apply_async(multiply, (x, y), callback=log_results,
error_callback=log_e)
pool.close()
pool.join()
print(results)
Old answer
This drove me crazy for a moment but then it made sense.
If I run it with multiply
changed like this:
def multiply(nums):
print("print")
return nums[0] * nums[1]
It runs fine. You said in the comments "I do not think the function multiply
is called in the first place." This is because there is a callback
specified but no error_callback
specified. The result of omitting an error callback is that your script is failing silently.
You could check this with:
import multiprocessing
results = []
def log_results(result):
print(result)
def log_e(e):
print(e)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name()}")
return x * y
pool = multiprocessing.Pool()
numbers = [[1,1], [2,2], [3,3]]
mapResult = pool.map_async(multiply, numbers, callback=log_results,
error_callback=log_e)
pool.close()
pool.join()
Which gives:
multiply() missing 1 required positional argument: 'y'
And with multiply
like so:
def multiply(nums):
return nums[0] * nums[1]
It then returns [1, 4, 9]
PS I am running Python 3.6.7
So your current code is actually failing because of this line:
print(f"Gets here for process name {multiprocessing.current_process().name()}")
it errors out as TypeError: 'str' object is not callable
, not because there is anything with the way you are calling multiply()
if you remove it:
import multiprocessing
results = []
def log_results(result):
results.append(result)
def multiply(x, y):
# print(f"Gets here for process name {multiprocessing.current_process().name()}")
return x * y
if __name__ == "__main__":
pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
print (f"Checking x {x} and y {y}")
pool.apply_async(multiply, (x, y), callback=log_results)
pool.close()
pool.join()
print(results)
It returns:
Checking x 1 and y 1
Checking x 2 and y 2
Checking x 3 and y 3
[1, 4, 9]
So if you actually isolate your print(f)
:
print(multiprocessing.current_process().name())
you get the error: TypeError: 'str' object is not callable
because
multiprocessing.current_process()
is actually a process object with name
as an attribute of the object which returns a string (thanks darkonaut) string. You are trying to call .name()
as a function, but it is an attribute.
So if you change your function to include .name
, instead of .name()
:
import multiprocessing
results = []
def log_results(result):
results.append(result)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name}")
return x * y
if __name__ == "__main__":
pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
print (f"Checking x {x} and y {y}")
pool.apply_async(multiply, (x, y), callback=log_results)
pool.close()
pool.join()
print(results)
You return:
Checking x 1 and y 1
Checking x 2 and y 2
Checking x 3 and y 3
Gets here for process name ForkPoolWorker-1
Gets here for process name ForkPoolWorker-2
Gets here for process name ForkPoolWorker-3
[1, 4, 9]
Which is what you desire.