what is web crawle code example

Example 1: Name the key term of what a "Crawler" is used to do

retrive and "index" webpages from the internet.

Example 2: web crawler

Intelligent goal-based agent