chemistry NLP corpus

It is a text corpus whose documents are chemistry: preprints, paragraphs and compound descriptions assembled for training and evaluating language models. The reference example here is the BASF-AI Chemical Data collection, which packages ten datasets.

Its text components are ChemRxiv-Papers, ChemRxiv-Paragraphs, ChemRxivRetrieval and dolma-chemistry-only, with dolma-chemistry-only at roughly 2.5 GB across 1.19M paragraphs. ChemRxivRetrieval is a retrieval benchmark - one of the ten datasets in the collection - so evaluation data ships alongside training text.

Distribution is parquet or JSON, and most sets are tagged cc-by-nc-4.0, which makes the collection's licensing a first-order purchase question: training inside a research setting is a different proposition from a commercial product.

Because domain text quality decides model behaviour, and provenance decides what you may do with the result. BASF-AI's collection draws its text from ChemRxiv preprints and a Dolma-derived chemistry paragraph set - named sources, not an undisclosed web scrape.

Licensing is the constraint to price in. Most sets in the collection are tagged cc-by-nc-4.0, so commercial deployment needs a separate rights path; the tag travels with the parquet and JSON files.

And evaluation is bundled. With a retrieval benchmark among the ten datasets, a team can measure chemistry retrieval quality on the same distribution it trains on - something a generic web corpus cannot offer.

1. Name the text sources. BASF-AI's collection is built from ChemRxiv preprints (ChemRxiv-Papers, ChemRxiv-Paragraphs) and a Dolma-derived chemistry paragraph set - demand that specificity from any corpus vendor.

1. Check scale per component. dolma-chemistry-only runs roughly 2.5 GB across 1.19M paragraphs; ask for per-component size, not collection totals.

1. Look for an evaluation split. ChemRxivRetrieval serves as a retrieval benchmark among the ten datasets - a corpus without one leaves you to build evaluation yourself.

1. Read the license tag before planning use. Most sets are tagged cc-by-nc-4.0 and ship as parquet or JSON - non-commercial terms shape the product roadmap, not just the budget.

Where to see it in context: diversified-chemicals data, specialty-chemicals data.

Entries that pair with chemistry nlp corpus:

• cc-noncommercial-license

• pubchem-derived-dataset

• smiles-notation