Glossary
Bulk Data Dump
A bulk data dump is a periodically released archive containing an entire dataset, letting heavy users mirror corpora without hammering query APIs. Open Library publishes monthly dumps of about 12.4 GB compressed for all types, and explicitly says bulk data should come from the dumps rather than its API.
What is a bulk data dump?
It is the publisher's answer to scale: instead of millions of API calls, take one (very large) file. Open Library's Monthly Data Dumps run '~12.4 GB compressed for all types in one dump', with editions alone around 9.2 GB; its terms prohibit bulk harvesting via API, stating 'Bulk data should come from the monthly dumps'.
Scale ranges widely across publishing sources. Project Gutenberg distributes bulk RDF at ~121 MB compressed, a CSV catalog near 20 MB, and all plain texts at roughly 10 GB compressed. OpenAlex publishes free quarterly snapshots of about 750 GB JSONL or ~780 GB Parquet - warehouse-scale by any standard.
Why do dumps matter when choosing a dataset?
They are often mandatory, not optional. Where terms redirect bulk consumers away from APIs - as Open Library does explicitly - scraping the query interface violates the operator's intent even when technically possible, and risks blocking.
Dump cadence sets freshness ceilings too: monthly snapshots mean up to a month of staleness, quarterly ones more. Size dictates infrastructure honestly - 750 GB JSONL demands object storage and columnar conversion, while Gutenberg's 20 MB CSV catalog fits a laptop. Matching dump choice to real analysis scope avoids paying storage for data you will never query.
How do you evaluate bulk data dumps in a source?
- Read the redirection clause. If terms say bulk belongs in dumps - as Open Library's do - honour it and skip API harvesting.
- Check cadence against freshness needs. Monthly editions dumps versus quarterly OpenAlex snapshots imply different staleness windows.
- Budget for size honestly. 12.4 GB, ~10 GB of texts or ~750 GB JSONL each imply very different storage designs.
- Pick the right internal format. RDF, JSONL, Parquet and CSV serve graph, lakehouse, columnar and spreadsheet workflows respectively.
- Verify licensing per component. Editions, texts and catalogs can carry different rights.
Related terms
- doi - persistent identifiers often attached to archived releases.
- open-access - the publishing model most dump publishers operate under.
- public-domain-ebook - much of what these dumps contain.
- work-edition-model - the record structure behind editions dumps.
Sector hub: publishing data.
Frequently asked questions
How big is an Open Library data dump?
About 12.4 GB compressed for all record types in one monthly dump, with editions alone accounting for roughly 9.2 GB.
Should I scrape an API or take the dump?
Take the dump when one exists - Open Library prohibits bulk harvesting via its API and directs bulk consumers to the monthly dumps explicitly.
Datasets containing this field
Datasets containing Bulk Data Dump
6 datasets carry bulk data dump in the catalog. Open one, count the fields, judge for yourself.
DOAJ API - Directory of Open Access Journals
bibjson.title · bibjson.alternative_title · bibjson.identifier …+19 more
Hugging Face Datasets Hub - Books Search
Every listing shows the field dictionary, sample rows, and coverage before you commit. API, files, or your warehouse. Daily, weekly, or hourly.
Get sample rows