Python: Writing to a single file with queue while using multiprocessing Pool
I took the accepted answer and simplified it for my own understanding of how this works. I am posting it here in case it helps someone else.
import multiprocessing
def mp_worker(number):
number += 1
return number
def mp_handler():
p = multiprocessing.Pool(32)
numbers = list(range(1000))
with open('results.txt', 'w') as f:
for result in p.imap(mp_worker, numbers):
f.write('%d\n' % result)
if __name__=='__main__':
mp_handler()
Multiprocessing pools implement a queue for you. Just use a pool method that returns the worker return value to the caller. imap works well:
import multiprocessing
import re
def mp_worker(filename):
with open(filename) as f:
text = f.read()
m = re.findall("x+", text)
count = len(max(m, key=len))
return filename, count
def mp_handler():
p = multiprocessing.Pool(32)
with open('infilenamess.txt') as f:
filenames = [line for line in (l.strip() for l in f) if line]
with open('results.txt', 'w') as f:
for result in p.imap(mp_worker, filenames):
# (filename, count) tuples from worker
f.write('%s: %d\n' % result)
if __name__=='__main__':
mp_handler()