How could I use requests in asyncio?
To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:
import asyncio
import requests
@asyncio.coroutine
def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = yield from future1
response2 = yield from future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
This will get both responses in parallel.
With python 3.5 you can use the new await
/async
syntax:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = await future1
response2 = await future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
See PEP0492 for more.
aiohttp can be used with HTTP proxy already:
import asyncio
import aiohttp
@asyncio.coroutine
def do_request():
proxy_url = 'http://localhost:8118' # your proxy address
response = yield from aiohttp.request(
'GET', 'http://google.com',
proxy=proxy_url,
)
return response
loop = asyncio.get_event_loop()
loop.run_until_complete(do_request())