Is there a multithreaded map() function?

Try the Pool.map function from multiprocessing:

http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers

It's not multithreaded per-se, but that's actually good since multithreading is severely crippled in Python by the GIL.


Try concurrent.futures.ThreadPoolExecutor.map in Python Standard Library (New in version 3.2).

Similar to map(func, *iterables) except:

  • the iterables are collected immediately rather than lazily;
  • func is executed asynchronously and several calls to func may be made concurrently.

A simple example (modified from ThreadPoolExecutor Example):

import concurrent.futures
import urllib.request

URLS = [
  'http://www.foxnews.com/',
  'http://www.cnn.com/',
  'http://europe.wsj.com/',
  'http://www.bbc.co.uk/',
]

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    # Do something here
    # For example
    with urllib.request.urlopen(url, timeout=timeout) as conn:
      try:
        data = conn.read()
      except Exception as e:
        # You may need a better error handler.
        return b''
      else:
        return data

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
    # map
    l = list(executor.map(lambda url: load_url(url, 60), URLS))

print('Done.')