Python Requests Module Not Getting Latest Data from Web Server

before your requests.get(), try adding a header:

import requests

url = "https://www.ncaa.com/scoreboard/basketball-men/d1/"

headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}   


response = requests.get(url, headers = headers)
html = response.text

My other suggestion would be to use:

url = 'https://data.ncaa.com/casablanca/scoreboard/basketball-men/d1/2019/01/26/scoreboard.json'

and use json package to read it. Everything is live and right there for you in a nice JSON format

Code

import json
import requests

url = 'https://data.ncaa.com/casablanca/scoreboard/basketball-men/d1/2019/01/26/scoreboard.json'
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}    

response = requests.get(url, headers = headers)

jsonStr = response.text

jsonObj = json.loads(jsonStr)

I checked, and the JSON object does return live scores/data. And all you need to do is change the date in the URL 2019/01/26 to get previous dates finished data for games.

---

EDIT - ADDITIONAL

This could help you pull out the data. Notice how I changed it to today's date to get the current data. It puts it in a nice dataframe for you:

from pandas.io.json import json_normalize
import json
import requests

url = 'https://data.ncaa.com/casablanca/scoreboard/basketball-men/d1/2019/01/27/scoreboard.json'
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}    

# Thanks to InfectedDrake wisdom, the following 3 lines that I previously had can be replaced by a single line. See below
#response = requests.get(url, headers = headers)
#jsonStr = response.text
#jsonObj = json.loads(jsonStr)

jsonObj = requests.get(url, headers = headers).json()

result = json_normalize(jsonObj['games'])

Try changing the user-agent in the request header to make it the same as your Google Chrome user-agent by adding this to your headers:

headers = {
    'User-Agent': 'Add your google chrome user-agent here'
}