Common Crawl Web Corpus

Common Crawl Web Corpus is the Interactive Home Entertainment catalog's view of the web itself: more than 300 billion crawled pages spanning fifteen-plus years, maintained by the Common Crawl project and carried here as queryable rows. Each crawl release - named CC-MAIN followed by year and week, 94-plus of them since 2008 - captures billions of pages during its window, and every capture becomes a record with its own timestamp, HTTP status, detected content type and payload digest. Cumulative volume sits in the petabytes.

The scale matters less than what it represents: a continuous, machine-readable memory of what the public web looked like, release after release, for nearly two decades. Pages that changed, moved or vanished remain here exactly as captured - which is why teams training language models, auditing a brand's digital footprint or studying how misinformation spreads start from this corpus rather than from the live web.

Get a sample of this dataset - name a domain set, content type or date window, and real rows come back before you commit to a feed.

Two real index records from the newest release in our August 2026 snapshot, shown exactly as they arrive:

url      :           urlkey    : io,itch)/
captured : 20260719035826            mime      : text/html
status   : 200                       length    : 18085
digest   : DRQK2TQJ5435MVZR5EGKINOYAGUURLU2
segment  : crawl-data/CC-MAIN-2026-30/segments/1783663952046.58/warc/…

url      :            status    : 301
redirect :           length    : 871
captured : 20260719035826            mime      : text/html
digest   : LGGT76I5HYSS6HQT36GPANELE4H7FWR7

Both were captured within the same second, and together they show why this corpus behaves like a database rather than a pile of HTML. The second row is a redirect: the crawler requested the plain-HTTP form of the same domain, got a 301, and the record preserves both the original address and its destination. Site migrations, domain consolidations and canonicalization choices are therefore reconstructable from rows alone. Note the urlkey column too - io,itch)/ is the SURT-canonicalized form of the address, which is what lets billions of captures sort and join consistently across releases.

Twelve verified columns per record, defined below. They fall into three groups. Identity fields (WARC-Target-URI, urlkey, WARC-Record-ID) say what was captured and make rows addressable and joinable. Provenance fields (WARC-Date, timestamp, filename, offset, WARC-Payload-Digest, WARC-IP-Address) say when and where the capture happened, down to the byte position inside the originating archive segment - so a single record is pinpointed even inside petabytes. Classification fields (WARC-Type, mime-detected, status) say what kind of exchange occurred, separating document fetches from redirects, errors and non-HTML payloads.

That last group carries most of the analytical weight. Filtering on status isolates live pages from dead links; filtering on mime-detected separates articles from images, PDFs and scripts; and the two together power churn studies - what share of a domain's 2019 pages still returned 200 in 2026 - without opening a single archive container.

Geography: global by construction. The crawler reaches every TLD and region its robots-compliant policy allows, so coverage reflects the public web rather than a country sample. Sites that opt out through robots directives are absent by design, and that absence pattern is steady enough to model explicitly.

Time: 94-plus named releases reach from 2008 to the present, and the newest in our August 2026 snapshot captured pages during July 2026. Because each release is preserved rather than overwritten, a question like "what did this site's pricing page say in March 2021" has a deterministic answer.

Granularity: one record per fetched URL per release, each carrying its own capture timestamp and payload digest. Cut by domain, content type, status code or time window and the cuts compose - a 2019-through-2026 slice of .de news sites returning 200s needs no custom parsing.

API, files, or your warehouse. Daily, weekly, or hourly. Match the channel to the workload: an API for applications that need pointed lookups on demand, files for bulk analysis runs, or direct delivery into your warehouse for teams joining web captures against their own tables. Cadence follows the question - hourly suits monitoring a fast-moving story, weekly suits footprint audits, and long-horizon model training usually wants a fixed historical cut plus a scheduled top-up.

Start smaller than a feed. Get a sample of this dataset scoped to the domains, content types or years you care about, verify the rows answer the question, then scale the delivery.

Language-model and NLP pipelines. The corpus remains the reference text layer for pre-training and domain-adaptive work precisely because it spans decades of natural web text rather than one platform's output. Payload digests make deduplication tractable at petabyte scale.

Brand and competitive footprint audits. Replaying past releases shows what a competitor's site advertised, priced or claimed in a given quarter - including pages since removed, which is often the interesting part.

Search and SEO research. Status codes, content types and canonicalized URL keys across billions of captures support churn, duplication and indexing-behavior studies at a scale no private crawl matches. Our comparison with Chrome UX Report (CrUX) covers when content-layer evidence beats performance telemetry.

Misinformation and media studies. Deleted pages persist as they were captured, so claims about what was published and when resolve against records instead of recollection - the property academic work cites most.

Due diligence on digital footprints. Acquirers and investors checking a company's historical web presence get timestamped, digest-verified evidence rather than screenshots.

• Data Scientists & ML Engineers - decades of digest-deduplicatable web text and status-labeled captures for pre-training, classification and churn models.
• Competitive Intelligence & Product Teams - replayed snapshots of competitor sites, including content competitors have since removed.
• Market Researchers & Consultants - category-level web-presence trends across fifteen-plus years of releases.
• Journalists, Academics & Students - citable, timestamped evidence of what was published online and when.
• Developers & Data-Product Builders - stable URL keys and record IDs behind archival, monitoring and citation products.
• Investors & Quants - digest-verified digital-footprint evidence during diligence on target companies.