The Data Format Landscape: One Catalog, 1,061 Labels, Four Pipelines

The by_format field reads less like a spec sheet than an inventory of what organizations actually ship. Of the 1,744 datasets Datadory catalogs, 734 (42.1%) offer CSV and 675 (38.7%) offer JSON, and together those two bracket almost every integration conversation. Excel-family spreadsheets arrive as a bloc rather than a single label: 569 datasets carry one of the three plain tags — xlsx (410), xls (97) and generic excel (62) — and once compound spellings such as "xlsx (codebooks, summary tables)" or "excel (zipped bulk)" are counted, the family totals 673 label mentions.

HTML is the fourth pillar at 572 datasets (32.8%), spanning raw pages, on-page tables and interactive dashboards, and PDF documents appear on 420 (24.1%). XML adds 287 (16.5%). Below the head sits a genuinely long tail: 71 datasets offer TSV, 66 GeoJSON, 54 plain txt and 36 PNG chart images, down to singletons such as EBCDIC, WikiText, WFDB heartbeat files and PC-Axis. The ranked picture below is the one to plan against.

Vendors describe identical containers in incompatible vocabularies. Plain "json" sits alongside "json (api)", "json_api", "json (rest api)", "json-stat", "json-ld" and "jsonp" in the same field, and statistical packages fragment the same way: SAS, SPSS, Stata and DDI spellings account for 43 label mentions across the catalog. Any coverage number computed before normalization will be wrong.

One record shows how fast labels multiply. The openFDA FAERS Adverse Event API lists json_api and json_bulk delivery next to pdf_reference, xlsx_reference and yaml_reference — a single source, five labels, two of which mean "JSON" to a normalizer. Its 20.7 million adverse-event reports arrive as roughly 113.7 GB spread across 1,767 quarterly JSON zip partitions, so the wrapper matters as much as the payload.

The practical rule: collapse labels into a family taxonomy — tabular, tree, document, geospatial, archive — before promising coverage to stakeholders. Field definitions are verified for 1,495 of the 1,744 datasets (85.7%), but format strings stay verbatim upstream, so the normalization step belongs in your pipeline, not the vendor's.

The catalog-wide hierarchy repeats in every industry debrief. Across the 159 industry briefs, CSV is named in 117 narratives and JSON in 104, with both appearing together in 78. Water utilities calls "CSV ... the dominant format". Oil and gas exploration & production reports that "Bulk Excel/CSV dominates access (12 of 18 primary datasets ship Excel or CSV)", with LAS, DLIS and SEGY reserved for subsurface releases.

Sector texture then bends the baseline. Aerospace-defense briefs say formats "lean XLSX, CSV and PDF". Regional banks receive FFIEC data as "Excel-compatible tab-delimited and XBRL files inside ZIP archives". Construction engineering pulls Socrata JSON/CSV/GeoJSON straight from city portals: NYC DOB job-application filings run about 2.72 million rows refreshed daily, and Chicago Building Permits adds about 845,000 rows across 122 columns. Biotechnology ignores office conventions altogether — its brief spans JSON, TSV, FASTQ, BAM, VCF, MAF, SDF, mmCIF and H5AD, the vocabulary of sequencing and structural biology rather than spreadsheets.

Four tiers cover the catalog end to end, and each has a distinct failure mode.

1. The tabular tier (roughly half the catalog). CSV, TSV and XLSX need delimiter sniffing, header-row handling and multi-sheet workbook parsing. Government microdata deepens the requirement: statistical-package labels (SAS, SPSS, Stata, DDI) total 43 mentions, concentrated in the releases electric-utilities and research-consulting-services buyers pull. Step zero is unzip-and-normalize — 237 datasets list ZIP outright, and archive and compression labels add up to roughly 382 mentions once .gz, .tar.gz and 7z variants are counted.

If you model or analyze: do not wait for Parquet. Only 86 sources ship it, so plan local conversion from CSV and JSON, and expect government microdata to arrive as SAS, SPSS or Stata files. The curated picks for data scientists sort sources by exactly these mechanics.

If you invest or quant-trade: banking and macro workflows are ZIP-wrapped mainframe text plus XBRL — the FFIEC pattern — so build quarterly bulk pulls with a parse layer, not live API taps.