FormationEval - Petroleum Geoscience LLM Benchmark

It is the first serious attempt at something like MMLU for the subsurface - a benchmark suite that asks a language model whether it actually knows petroleum geoscience, petrophysics and drilling engineering rather than just English.

Three public tracks make up 1,632 items in roughly 13.6 MB of JSON. Track one holds 505 author-written multiple-choice questions across seven domains - Petrophysics, Petroleum Geology and Drilling Engineering among them - and is the set used to evaluate 72 models in December 2025. Track two adds 1,027 question-answer pairs imported from the DISKOS-QA corpus on 17 March 2026. Track three contributes 100 MCQs drawn from the Study Guide for the SPE Petroleum Engineering Certification Examination (4th edition).

What makes the suite unusual is not the count but the self-description. Every item states its own lineage: which track produced it, which discipline and topic it examines, how hard it is, where its material came from under what license, and whether it risks having been seen during training. Most domain question banks ship as bare text files with an answer sheet; this one ships as a database.

Two captured items, one from each end of the suite's provenance spectrum:

# track file formationeval_v0.1.json - authored MCQ item
id               : formationeval_v0.1_petrophysics_logging_principles_001
version          : formationeval_v0.1
domains          : Petrophysics | Petroleum Geology
topics           : Well Logging History | Measurement Principles
difficulty       : easy
language         : en
answer_key       : D
derivation_mode  : concept_based

metadata:
  calc_required      : false
  contamination_risk : medium

# track file formationeval_spe_mcq_v0.3.json - imported SPE certification item
id               : formationeval_v0.3_spe_mcq_001
domains          : Drilling Engineering
topics           : (empty on imported tracks)
difficulty       : unknown
question         : Which of the following factors has the LEAST impact
                   on casing seat selection?
choices          : A) Rate of Penetration   B) Pore pressure
                   C) Kick tolerance        D) Fracture gradient
answer_key       : A

Read them side by side and the suite's whole design shows up. The authored item arrives fully described - two domains, two topics, a difficulty grade, a derivation route, and a contamination flag admitting the question might be circulating in training corpora. The imported item keeps its question, choices and key but carries empty topics and an unknown difficulty, which is honest: nobody regraded 100 imported certification questions by hand. Datadory ships both shapes intact, so your harness can filter on description richness instead of discovering it item by item.

Fifteen documented fields carry every item, split between a grading spine and an audit spine.

The grading spine is choices read against answer_index and answer_key - the same correct position expressed as both a zero-based integer and a letter, so evaluation code written either convention consumes it without a translation layer. question, rationale and difficulty complete what a scorer needs; rationale explains why the keyed answer wins, which turns the suite into teaching material as well as a test.

The audit spine is sources plus derivation_mode. Each item names its source titles, DOIs, licenses, attribution and retrieval dates inside a structured array, and states whether it was concept-authored, built from open-licensed material, or imported wholesale from another benchmark. metadata.calc_required and metadata.contamination_risk close it out - the first splits arithmetic evals from recall evals, the second grades the odds the item text already sits in training data.

Four more artifact groups ride alongside the items themselves and therefore fold under additional fields on request: suite manifests, leaderboard and analysis outputs, QA-track format metadata, and third-party notice detail for imported batches.

Geographically this is a discipline play, not a map: seven domains spanning global petroleum practice, with cited literature dating 2007 to 2025 and no basin weighting claimed. A question about logging principles grades a model the same in Houston as in Stavanger.

Temporally the suite is young and dated precisely: the authored v0.1 track carried the December 2025 evaluation of 72 models, the DISKOS-QA import landed 17 March 2026, and the SPE track stands at v0.3. No forward cadence is promised - the project describes itself as self-funded and single-person - so treat further growth as possible rather than scheduled.

At grain level the unit is the individual exam item: 1,632 JSON objects across three track files, each tagged along five axes (domain, topic, difficulty, derivation mode, contamination risk). The distribution matters more than usual here - the 505 authored items are the evaluated core, while the 1,027 DISKOS-QA pairs and 100 SPE items are larger but publicly unscored, so any capability claim should say which portion it rests on.

API, files, or your warehouse. Daily, weekly, or hourly.

Your cadence is your call regardless of how the upstream suite moves - a static, versioned benchmark does not churn, so most teams take a file drop per track and refresh only when a new version lands. Every delivery ships the complete field dictionary above, the captured sample rows, and the track-level manifest so your evaluation runs pin to exact versions.

If your harness expects a particular shape - one row per item flattened into columns, or nested exactly as the suite publishes it - say so in the sample request. Flattening the sources array and metadata flags into joinable columns is the most common cut we make, because it lets dashboards report scores per domain and per contamination band without a parsing step.

• Data scientists and ML engineers score language models on subsurface knowledge where general-purpose suites fall silent - and use the contamination_risk flag to separate models that know geology from models that memorized a textbook's question bank.
• Developers building AI products for energy regression-test domain drift: rerun the same pinned track whenever an underlying model updates, and a silent degradation in drilling-engineering answers surfaces before customers find it.
• Training organizations and academics reuse the rationale field as ready-made explanations, turning an evaluation set into course material with the same objects that grade it.
• Analysts tracking AI capability claims check vendor benchmarks against a dated public run - 72 models scored in December 2025 - instead of trusting marketing decks.

Data scientists and ML engineers lead the tagging at relevance 3, because a domain-graded, provenance-explicit question bank is the scarce input for any claim about petroleum-domain AI competence. Developers and builders match them - typed items with dual-format answer keys drop straight into harnesses; see developers builders use cases.

Journalists, academics and students rank next at relevance 2: a citable dated run over a published suite beats assembling ad-hoc quizzes to probe a model's geology. Competitive intelligence and product teams close it out, mapping the item taxonomy onto product roadmaps for AI in drilling and subsurface work; see data scientists use cases for the modeling angle.

If your role involves asserting that a model knows oil and gas, this is the shortest path from assertion to evidence in the catalog.

Within oil & gas drilling data, almost nothing else competes here - because almost nothing else is about models at all. The slice is dominated by operational registries: BSEE Well API covers 57,482 offshore well records at a 9/10 Datadory score, and the Texas Railroad Commission data sets hold every Texas well and lease at 9/10. Those answer what happened in the ground; this record answers what a model knows about the ground.

The nearest methodological neighbour is DataDRILL, scored 8, which ships two CSVs of real-time drilling sensor series for training predictive models. They are complements, not rivals: DataDRILL trains what a model predicts from rig telemetry; FormationEval grades what a model already knows from text. Catalog-wide the pattern holds - among 1,744 datasets Datadory catalogs, LLM benchmark records are rare birds, and petroleum-domain ones effectively unique to this record.

Three things worth settling upfront.

First, know which portion you are evaluating. The published leaderboard covers the 505-question authored v0.1 track only; the 1,127 imported items are newer and publicly unscored. If your comparison needs the evaluated core, pin v0.1; if it needs volume, take all three tracks and budget for running them yourself.

Second, use the contamination flag honestly. Items flagged higher risk may live in training data, so a strong score there is weaker evidence than the same score on low-risk items. We deliver the flag as a typed column so you can report results per band rather than blended.

Third, expect irregular growth. Release cadence is unstated and the project runs on one person's funding, so build against versioned filenames and manifests rather than assuming new tracks arrive on schedule. On quality: Datadory scores this record 6/10 on its documentation-and-reliability rubric - below the 7.81 catalog mean - with deductions landing on freshness cadence and the unevaluated import majority, not on field documentation, which is verified.