What's the best way to download file using urllib3
Your code snippet is close. Two things worth noting:
If you're using
resp.data
, it will consume the entire response and return the connection (you don't need toresp.release_conn()
manually). This is fine if you're cool with holding the data in-memory.You could use
resp.read(amt)
which will stream the response, but the connection will need to be returned viaresp.release_conn()
.
This would look something like...
import urllib3
http = urllib3.PoolManager()
r = http.request('GET', url, preload_content=False)
with open(path, 'wb') as out:
while True:
data = r.read(chunk_size)
if not data:
break
out.write(data)
r.release_conn()
The documentation might be a bit lacking on this scenario. If anyone is interested in making a pull-request to improve the urllib3 documentation, that would be greatly appreciated. :)
The most correct way to do this is probably to get a file-like object that represents the HTTP response and copy it to a real file using shutil.copyfileobj as below:
url = 'http://url_to_a_file'
c = urllib3.PoolManager()
with c.request('GET',url, preload_content=False) as resp, open(filename, 'wb') as out_file:
shutil.copyfileobj(resp, out_file)
resp.release_conn() # not 100% sure this is required though