Open Library Monthly Dump Data: Dataset Structure and Field Coverage

Each dump line holds one record - edition, work, author, redirect or list - as a JSON payload per row, which makes parsing a streaming job rather than a database restore. The catalog behind the dumps is large: a broad search query returned numFound 7,723,767 works during testing, and the homepage states more than 3M books are borrowable free.

The current all-types dump weighs about 12.4 GB compressed, and the editions file alone accounts for roughly 9.2 GB of that. If you also want revision history, the 'complete' variant runs about 29.6 GB compressed and additionally includes every past revision of every record.

Beyond the core bibliographic types, the dumps page lists separate ratings, reading-log, lists, covers and wikidata files, so reader-behavior signals ride along with the metadata instead of requiring a second collection effort. Formats are TSV (one line per record), JSON payloads inside those lines, and txt.gz archives.

A working pipeline needs five steps, and none of them involve requesting credentials:

1. Decompress each txt.gz streamingly (gzip -dc | python) rather than expanding to disk; every line is a self-contained JSON payload keyed by record type.

1. Split by type first: load editions (~9.2 GB compressed) into an ISBN-keyed table, works into a title-level table, authors separately, then keep redirects to resolve later merges.

1. Join the side files - ratings, reading-log, lists, covers, wikidata - onto your edition and work keys so engagement signals land in the same warehouse as the metadata.

1. For anything newer than last month's snapshot, fall back to the keyless Search & Works/Editions API at 1 req/s anonymous (3 req/s with an identified User-Agent and email), never at bulk-harvest volumes.

The dump gives you metadata; three sibling sources give it payload. Project Gutenberg publishes 79,228 public-domain ebooks whose RDF, CSV and MARC catalogs join to Open Library records on title and author. The Internet Archive Digital Library's Text & Books Collection exposes OCR text files and PDFs over 40M+ scanned items through an advancedsearch Solr API - 4,093,835 items sat in its americana collection alone on 2026-08-21. For model training, Hugging Face's Datasets Hub lists about 770 book-tagged corpora, from LoC-PD-Books (~140,000 books, ~8 billion words) downward.