Using requests module, how to handle 'set-cookie' in request response?

There's an included class called a session which automatically handles this sort of thing for you. You can create an instance of it, and then call get and set right on that instance instead.

import requests

URL1 = 'login prompt page'
URL2 = 'login submission URL'

session = requests.Session()

r = session.get(URL1)
r2 = session.post(URL2, data="username and password data payload")

Another way that has worked for me (without using session objects) is the following (tested in v2.18.4).

jar = requests.cookies.RequestsCookieJar()
response1 = requests.get(some_url, cookies=jar) # or post ...
jar.update(response1.cookies)
response2 = requests.get(some_other_url, cookies=jar) # or post ...

Note that the above code will fail in the presence of redirects which are handled transparently by the Requests library. In such a case, you also have to update your jar with the cookies sent in the redirect responses. E.g. by doing something like the following:

if (response.history): # we have to add to the cookie jar, the cookies sent by the server in intermediate responses 
    for historicResponse in response.history:
        jar.update(historicResponse.cookies)

Ignore the cookie-jar, let requests handle cookies for you. Use a Session object instead, it'll persist cookies and send them back to the server:

with requests.Session() as s:
    r = s.get(url1)
    r = s.post(url2, data="username and password data payload")