Python requests call with URL using parameters
you will need to URL encode the URL you are sending to the API.
The reason for this is that the ampersands are interpretted by the server as markers for parameters for the URL https://extraction.import.io/query/extractor/XXX?
This is why they are getting stripped in the url:
http://www.example.co.uk/items.php?sortby=Price_LH
Try the following using urllib.quote(row_dict['url'])
:
import requests
import json
import urllib
row_dict = {
'url': u'http://www.example.co.uk/items.php?sortby=Price_LH&per_page=96&size=1%2C12&page=35',
'crawler_id': u'zzz'}
url_call = 'https://extraction.import.io/query/extractor/{0}?_apikey={1}&url={2}'.format(
row_dict['crawler_id'], auth_key, urllib.quote(row_dict['url']))
r = requests.get(url_call)
rr = json.loads(r.content)
The requests
library will handle all of your URL encoding needs. This is the proper way to add parameters to a URL using requests
:
import requests
base_url = "https://extraction.import.io/query/extractor/{{crawler_id}}"
params = dict()
params["_apikey"] = "xxx"
params["url"] = "http://www.example.co.uk/items.php?sortby=Price_LH&per_page=96&size=1%2C12&page=35"
r = requests.get(base_url, params=params)
print(r.url)
An arguably more readable way to format your parameters:
params = {
"_apikey" : "xxx",
"url" : "http://www.example.co.uk/items.php?sortby=Price_LH&per_page=96&size=1%2C12&page=35"
}
Note that the {{crawler_id}}
piece above is not a URL parameter but part of the base URL. Since Requests is not performing general string templating something else should be used to address that (see comments below).