What we measure when a new AI model ships
Every frontier release gets the same thirty claims, the same prompt, and no internet. Then we publish what it invented.
When a new model launches, the numbers you can find are the ones the vendor chose to publish. We run our own test instead: an identical set of claims put to every model with no search and no retrieval, scored by whether the DOIs it produced actually exist, and, from August 2026, by whether its verdict on a claim bends when a user pushes back. Each page below carries the result for one model version, alongside the model it replaced.
Every comparison stays inside one vendor's line and one tier. A Flash-tier model is measured against the Flash model before it, a flagship against the flagship before it. Ranking a cheap model against a frontier one and calling the gap a finding is the kind of number this benchmark exists to catch.
Releases we have measured
Does Gemini 3.7 Flash fabricate fewer citations than Gemini 3.6 Flash?
Three runs of the same test disagreed with each other about which model was better. That is the finding.
Does Grok 4.6 fabricate fewer citations, or bend less under pressure, than Grok 4.5?
Two measurements on one claim set: invented DOIs, and verdicts that moved when a user pushed back. Neither separated the two generations.
Does Claude Fable 5.1 fabricate fewer citations than Claude Fable 5?
Thirty claims, three runs, every DOI resolved against Crossref. The most expensive model we can buy, measured against the one it replaced.
Does Gemini 3.8 Flash fabricate fewer citations than Gemini 3.7 Flash?
The newer model invented more sources in all three runs, and that still is not enough to call it worse.
Does GPT-6 Astra fabricate fewer citations than GPT-5.6 Sol?
The lowest fabrication rate we have measured for an OpenAI model, and the steadiest numbers any model has posted here.
Why a page per release
A model launch produces a burst of questions and almost no evidence, and by the time anyone has run a real test the attention has moved on. These pages are the test, kept at a permanent address so the answer is still here in six months, when the question has changed from what shipped this week to which of these models should I actually trust.
The current standings put every model we have measured on one page, with the caveats that come with it.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.