Common Crawl

Common Crawl

Type	501(c)(3) non-profit
Headquarters	San Francisco, California, USA; Los Angeles, California, USA
Founder(s)	Gil Elbaz
Key people	Peter Norvig, Nova Spivack, Carl Malamud, Kurt Bollacker Joi Ito
Website	commoncrawl.org
Available in	English

Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of 145 TB of data from 1.81 billion webpages as of August 2015. It completes four crawls a year.

Common Crawl was founded by Gil Elbaz. Advisors to the non-profit include Peter Norvig and Joi Ito. The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.

History

Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.

The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July of that year. Common Crawl's archives had only included .arc files previously.

In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012. The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."

In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler. Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.

History of Common Crawl data

The following data have been collected from the official Common Crawl Blog

Crawl Date	Availability date	Size in TB	Billion of pages	Comments
November 2015	December 2015	151	1.82
September 2015	November 2015	106	1.32
July 2015	August 2015	145	1.81
June 2015	July 2015	131	1.67
May 2015	July 2015	159	2.05
April 2015	May 2015	168	2.11
March 2015	May 2015	124	1.64
February 2015	March 2015	145	1.9
January 2015	March 2015	139	1.82
December 2014	January 2015	160	2.08
November 2014	December 2014	135	1.95
October 2014	November 2014	254	3.7
September 2014	November 2014	220	2.8
August 2014	September 2014	200	2.8
July 2014	August 2014	266	3.6	Crawl dataset available every Month
April 2014	July 2014	183	2.6
March 2014	March 2014	223	2.8	First Nutch crawl
January 2014	January 2014	148	2.3
November 2013	November 2013	102	2	Data in Warc file format
July 2012	July 2012	-	-	Data in Arc file format
January 2012	January 2012	-	-	Public Data Set of Amazon Web Service
November 2011	November 2011	40	5	first availability on Amazon

Norvig Web Data Science Award

In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux. The award is named for Peter Norvig who also chairs the judging committee for the award.

External links

• Common Crawl in California, United States
• Common Crawl GitHub Repository with the crawler, libraries and example code
• Common Crawl Discussion Group
• Common Crawl Blog