Ask ChatGPT a regulatory question and the answer comes back fast and self-assured. Then you check it: a regulation number that does not exist, a superseded edition, a draft quoted as binding. After enough of these, the verdict feels obvious: ChatGPT is not ready for regulatory work.
It is the wrong verdict. The GPT models people use are perfectly capable of regulatory reasoning. What fails them is reach: a general model answers from a frozen snapshot of the web, with no way to open the actual text of a regulation or to know whether it is in force today. Give it that text, and it stops guessing.
That text is what Obsidian supplies. We put the GPT models, GPT-5.4-nano, GPT-5.4-mini and GPT-5.5, through hundreds of complex regulatory tasks across ESG, chemicals and life sciences, each handled alone and connected to Obsidian.
ChatGPT is inaccurate for regulatory work
Alone, the three GPT models averaged 54 out of 100. Connect them to Obsidian and the average climbs to 95. The best pairing, gpt-5.4-mini, reached 95.5. The models did not change between those two numbers. Only the data in front of them did.
The standout is the smallest one. gpt-5.4-nano, at $0.463 per million tokens, climbs from 42 to 96 once connected, into the band of models many times its price. On regulation the data layer outweighs the model size: you hand the data to a small, fast model rather than buying a bigger one, and the per-model table shows the same convergence across all three.
ChatGPT cannot point you to the official source
Accuracy is only half of it. Connected to Obsidian, a GPT answer shows its work: the instrument, its exact reference and edition, the legal status, and a direct link to the official document, often the source PDF. Alone, you get a plausible citation to verify yourself. Connected, it arrives already checkable, which is the part a compliance workflow actually needs.
An answer with the tier-0 source attached is one you can forward to an auditor without re-checking it. That is the difference between a draft a model imagined and an obligation you can act on.
ChatGPT hallucinates
We broke every GPT answer into its individual factual claims and checked each against the official source. The gap between the two grounded-claim numbers above is the dangerous kind of error removed: the confident statement with nothing behind it. The ungrounded remainder is added context around the source, not fabricated references.
The full data, for the purists
Every model, both conditions. "Alone" is the model with no data layer; "with Obsidian" is the same model connected. Accuracy is a 0 to 100 score from a blind judge against human-verified ground truth. "Grounded claims" is the share of the answer's atomic factual claims that trace back to the official source, alone versus with Obsidian.
| # | Model | Tier | Acc. alone | Acc. + Obsidian | Lift | Cites source | Status correct | Grounded claims (alone → +Obs) | Latency | Speed | Price /1M | Cost / question |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | gpt-5.4-mini | mid | 69.5 | 95.5 | +26.0 | 96% | 100% | 38% → 96% | 1.25s | 84 tok/s | $0.7 | $0.000966 |
| 2 | gpt-5.4-nano | light | 42.5 | 95.5 | +53.0 | 94% | 99% | 28% → 96% | 1.42s | 83 tok/s | $0.463 | $0.000551 |
| 3 | gpt-5.5 | advanced | 49.8 | 94.4 | +44.6 | 96% | 100% | 44% → 96% | 4.89s | 42 tok/s | $11.25 | $0.0167 |
Pooled across every answer, even the smallest GPT connected to Obsidian beats the frontier GPT answering alone, for less than a cent a question.
How we measured it
- Three GPT models: GPT-5.4-nano, GPT-5.4-mini, GPT-5.5.
- Hundreds of complex regulatory tasks across ESG (CSRD, the ESRS, the EU Taxonomy, SFDR), chemicals (REACH, the UN GHS, the global conventions) and life sciences (the ISO and IEC medtech standards, ICH, IMDRF), each tied to its official source.
- Two conditions: GPT alone, and GPT connected to Obsidian.
- A blind judge scores each answer against human-verified ground truth; grounded claims come from a separate per-claim check.
Make ChatGPT the model in row one
Connect Obsidian to ChatGPT and every regulatory answer comes back with its official source, date and legal status, often for less than a cent a question. Free tier, two-minute setup.
Explore the Obsidian data layerWhat this means
The GPT you already use, given verified regulatory data, answers with the precision of a specialist and the receipts of an auditor. The background is here too: why AI hallucinates on regulatory questions, what tier-0 regulatory data is, and the idea of agentic regulatory intelligence. The full cross-provider results are in the regulatory AI benchmark. To test it on your own questions, connect the Obsidian regulatory data layer.