How do you use an HTTP/HTTPS proxy with boto3?
If you user proxy server does not have a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"
if you user proxy server has a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://user:[email protected]:port"
os.environ["HTTPS_PROXY"] = "https://user:[email protected]:port"
As of at least version 1.5.79, botocore accepts a proxies
argument in the botocore config.
e.g.
import boto3
from botocore.config import Config
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))
boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource
botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config
This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:
import botocore.endpoint
def _get_proxies(self, url):
return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3
Apart from altering the environment variable, I'll present what I found in the code.
Since boto3 uses botocore, I had a look through the source code:
https://github.com/boto/botocore/blob/66008c874ebfa9ee7530d944d274480347ac3432/botocore/endpoint.py#L265
From this link, we end up at:
def _get_proxies(self, url):
# We could also support getting proxies from a config file,
# but for now proxy support is taken from the environment.
return get_environ_proxies(url)
...which is called by proxies = self._get_proxies(final_endpoint_url)
in the EndpointCreator
class.
Long story short, if you're using python2 it will use the getproxies
method from urllib2 and if you're using python3, it will use urllib3.
get_environ_proxies
is expecting a dict containing {'http:' 'url'}
(and I'm guessing https
too).
You could always patch
the code, but that is poor practice.