what is a web crawler code example

Example 1: what is webworker

web worker
Web Workers are a mechanism to create a separate thread 
in parallel to your main JS program thread. 
can only communicate with each other through normal async events 
always abide by the event-loop one-at-a-time behavior 
required by run-to-completion.

Example 2: Name the key term of what a "Crawler" is used to do

retrive and "index" webpages from the internet.

Example 3: web crawler

Intelligent goal-based agent