Free Large datasets to experiment with Hadoop

This is a collection of 189 datasets for machine learning (which is one of the nicest applications for hadoop g): http://archive.ics.uci.edu/ml/datasets.html

One obvious source: the Stack Overflow trilogy data dumps. These are freely available under the Creative Commons license.

Few points about your question regarding crawling and wikipedia.

You have linked to the wikipedia data dumps and you can use the Cloud9 project from UMD to work with this data in Hadoop.

They have a page on this: Working with Wikipedia

Another datasource to add to the list is:

• ClueWeb09 - 1 billion webpages collected between Jan and Feb 09. 5TB Compressed.

Using a crawler to generate data should be posted in a separate question to one about Hadoop/MapReduce I would say.