What we measure when a new AI model ships
Every frontier release gets the same thirty claims, the same prompt, and no internet. Then we publish what it invented.
When a new model launches, the numbers you can find are the ones the vendor chose to publish. We run our own test instead: an identical set of claims put to every model with no search and no retrieval, scored by whether the DOIs it produced actually exist. Each page below carries the result for one model version, alongside the model it replaced.
Every comparison stays inside one vendor's line and one tier. A Flash-tier model is measured against the Flash model before it, a flagship against the flagship before it. Ranking a cheap model against a frontier one and calling the gap a finding is the kind of number this benchmark exists to catch.
Why a page per release
A model launch produces a burst of questions and almost no evidence, and by the time anyone has run a real test the attention has moved on. These pages are the test, kept at a permanent address so the answer is still here in six months, when the question has changed from what shipped this week to which of these models should I actually trust.
The current standings put every model we have measured on one page, with the caveats that come with it.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.