crawling sites residential gateway code example

Example: Name the key term of what a "Crawler" is used to do

retrive and "index" webpages from the internet.