Asyncio Making HTTP Requests Slower?

You are waiting for each request to finish before you start the next one. So you have the overhead of the event loop with no benefits.

Try this:

import asyncio
import functools
import requests
import time

ts = time.time()
loop = asyncio.get_event_loop()

@asyncio.coroutine
def do_checks():
    futures = []
    for i in range(10):
        futures.append(loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3)))

    for req in asyncio.as_completed(futures):
        resp = yield from req
        print(resp.status_code)

loop.run_until_complete(do_checks())
te = time.time()
print("Version A: " + str(te - ts))

ts = time.time()
for i in range(10):
    r = requests.get("http://google.com", timeout=3)
    print(r.status_code)
te = time.time()
print("Version B:  " + str(te - ts))

This is what I get when i run it:

$ python test.py 
200
...
Version A: 0.43438172340393066
200
...
Version B: 1.6541109085083008

Much faster, but really this is just spawning threads and waiting for the http library to finish, you don't need asyncio to do that.

You might want to checkout aiohttp as it was built for use with asyncio. requests is a fabulous library, but it is not made for asyncio.

Just for completeness, here is a really fast asyncio implementation

import aiohttp
import asyncio
import time

async def main(n):
    ts = time.time()
    session = aiohttp.ClientSession()
    fs = [session.get('http://google.com') for _ in range(n)]
    for f in asyncio.as_completed(fs):
        resp = await f
        print(resp.status)
        await resp.release()
    await session.close()
    te = time.time()
    print("Aiohttp version:  " + str(te - ts))

loop = asyncio.get_event_loop()
loop.run_until_complete(main(10))
loop.close()

The code is python 3.5 and higher.

~> python asyncioreq.py
200
...
Aiohttp version:  0.15974688529968262

Hope somebody can use it ;)

Building on @brunsgaard's answer: you can go a step further with aiohttp and gathering your results with asyncio.gather(). You can then take the responses from your requests and handle them.

import aiohttp
import asyncio
import time

async def main(n):
    start = time.time()
    session = aiohttp.ClientSession()
    jobs = [session.get('http://google.com') for _ in range(n)]
    done_jobs = await asyncio.gather(*jobs)
    for done_job in done_jobs:
        print(done_job.status)
    session.close()
    end = time.time()
    print("Time:  " + str(end - start))

loop = asyncio.get_event_loop()
loop.run_until_complete(main(10))
loop.close()