requests, cannot assign requested address, out of ports?
The way I resolved my issue was to use the requests.Session
class where I would reuse the same connection/session for each call in a given process.
Contrived example:
import requests
for line in file:
requests.get('http://example.com/api?key={key}'.format(key=line['key']))
becomes
import requests
with requests.Session() as session:
for line in file:
session.get('http://example.com/api?key={key}'.format(key=line['key']))
These questions had some related advice:
Repeated POST request is causing error "socket.error: (99, 'Cannot assign requested address')" Python urllib2: Cannot assign requested address
I too faced similar issue while executing multiple POST statements using python's request library in Spark. To make it worse, I used multiprocessing over each executor to post to a server. So thousands of connections created in seconds that took few seconds each to change the state from TIME_WAIT and release the ports for the next set of connections.
Out of all the available solutions over the internet that speak of disabling keep-alive, using with request.Session() et al., I found this answer to be working. You may need to put the header content in a separate line outside the post command though.
headers = {
'Connection': 'close'
}
with requests.Session() as session:
response = session.post('https://xx.xxx.xxx.x/xxxxxx/x', headers=headers, files=files, verify=False)
results = response.json()
print results