Every product built on AI has a list of places where somebody typed a model name into a config file. Ours had six. Four free tools, the step that turns four models' output into a report, and the classifier that decides how a submission gets verified. One of those six had ever been benchmarked. The other five were running a model ID that someone picked once and nobody revisited, and three of them pointed at a preview build that had been superseded twice over.
So we tested them. 34 fixtures, four to five candidates per job, 227 graded calls, 64 cents. Five of the six choices changed. What follows is the method, the tables, and the parts that did not flatter us — including a test case we had labelled wrong and only noticed because all five models disagreed with it. If you want the sourced roundup of what other benchmarks measured in 2026, that is over here; this is our own data about our own product.
What we changed
Each job was run against its own fixture set with its own scorer, using the real prompt, the real token limit and the real response format. The only thing that varied was the model.
Six jobs, before and after
| Job | Was running | Now runs | Why |
|---|---|---|---|
| Claim checker | gemini-3-flash-preview | gemini-3.5-flash-lite | Perfect score on nine labelled claims, fastest of the five, second cheapest. |
| Pre-mortem generator | gemini-3-flash-preview | gemini-3.5-flash-lite | The only candidate that came in under the latency gate and finished every fixture. |
| Sycophancy detector | gemini-3-flash-preview | gpt-5.6-luna | Four-way tie on quality within 0.8 points. Price decided it. |
| Report consolidator | gemini-3-flash-preview | gpt-5.6-luna | Three models tied on a clean 100. This one is 2.7x cheaper than the model it replaces. |
| Mode classifier | gemini-3-flash-preview | gpt-5.6-luna | 96.7 against the incumbent's 93.3 over eight cases run three times each, at 2.5x lower cost. |
| AI humanizer | gpt-5.6-luna | gpt-5.6-luna | Unchanged. Sonnet 5 edits about four points better and costs 17.5 times more. |
The one job we had benchmarked before is the one that did not move. That is the whole finding in a sentence. A model choice nobody has measured is not a decision, it is a leftover, and ours had been sitting there long enough that the ID pointed at a preview build two releases behind.
Every table, the raw per-call results and the fixtures are in the repository, and the whole study re-runs with one command. A test in our build fails if any shipped model stops matching what the committed evidence picks, which is the part that stops this drifting again.
The rule, written before the tables
The failure mode of a benchmark like this is reading four tables and picking your favourite. So the selection rule was written down first, encoded in code rather than memory, and applied identically to all six jobs.
A model that failed any fixture is out
Whatever it scored on the rest. A tool that dies on one run in four is not a candidate, and averaging that failure away is how you ship it anyway.
A model averaging over ten seconds is out
This is a product constraint, not a measurement. These pages promise an answer in seconds and they are most people's first contact with us.
Take everything within five points of the best survivor
At four to nine fixtures per job, a two-point gap is not a finding. Treating it as one would be exactly the overclaiming these tools exist to catch.
Pick the cheapest of those
Once quality is indistinguishable, price is the only honest tiebreak left.
Steps one and two are judgement calls and we would happily argue about the ten-second line. Steps three and four are not negotiable. The tolerance is what stops a decimal place from making a decision it has not earned, and on two of the six jobs the tolerance is the only reason we did not report a winner that did not exist.
No model grades a model
Nothing here is judged by an LLM. Every score is computed from the fixture and the response, which means it is reproducible, arguable, and cannot quietly inherit a judge's preference for its own family. It also meant designing a real checkable rule for each job rather than asking a model whether the output looked good.
Labelled ground truth where it exists
The claim checker gets nine claims with settled verdicts, three each of true, partly true and false. An exact match scores full marks, calling a false claim partly true scores half, and calling it true scores zero. A wrong verdict held with high confidence loses another fifteen points, because confident and wrong is worse than uncertain and wrong.
Quote grounding
The sycophancy detector quotes the sentence it is flagging. We check that the quote actually appears in the submitted text. A detector that invents the line it is objecting to is worse than one that finds nothing, and it is the only error in that output a reader cannot catch by reading.
Arithmetic the prompt asked for
The consolidator's prompt spends a paragraph telling the model to count claim statuses from its own output and state them. Whether it did is arithmetic, so we check the arithmetic. This turned out to be the only metric that separated the top models from the rest on that job.
Grounding in the source
A pre-mortem risk has to reuse words from the plan it is about. A risk that mentions nothing specific would fit any submission, which is what a template does. The consolidator gets the same treatment against the reviewer text it is summarising.
This is the same principle the product runs on. A single model marking its own work inherits every blind spot it already had, which is why TrueStandard checks a draft with four models from different labs and shows you where they disagree rather than handing you one confident answer. You can see the shape of it on the free claim checker, or read why a model cannot check its own work.
Half the tables decided nothing
This is the part a benchmark write-up usually leaves out. On the claim checker every model scored between 90.6 and 100. On the sycophancy detector, between 98.2 and 99.0. On the consolidator, three of five tied on a clean 100.
Those are ceilings, not rankings. The fixtures were not hard enough to tell good models apart, and three of our six decisions were made on price and speed with quality contributing nothing.
That is worth saying plainly, because the tempting move is to report the ordering anyway. A 0.8-point gap across six passages is noise wearing a decimal point, and a table that presents it as a result is doing the thing we built the sycophancy detector to detect. Harder fixtures would separate these models. We have not written them yet.
The practical read: for well-specified extraction and classification work, the current generation of fast models is close enough that price and latency are the real variables. That is a useful finding on its own, and it is the opposite of what a vendor ranking would have told you.
Three things it found that we were not looking for
The benchmark was built to choose models. Most of its value turned out to be elsewhere.
The four-model run was billing like a one-model run
Paid accounts run our tools across four models instead of one. Credits are charged per thirty thousand tokens, and a four-model tool run burns about five thousand, so it rounded to the same single credit a one-model run costs. The upgrade was free and had been since the day it shipped. It now floors at four credits, matching the pre-flight check.
A model cannot finish a pre-mortem at the token limit we set
Claude Sonnet 5 hit finish_reason: length on all four pre-mortem fixtures, truncated mid-array at exactly 2,048 completion tokens. That is a configuration incompatibility rather than a quality verdict, and it is worth separating the two: we did not measure what it scores with room to finish.
Reasoning tokens come out of the same budget
Google's endpoints reason whether you ask them to or not, and those tokens spend the same allowance as the answer. On one job Gemini used 711 to 1,735 tokens thinking before writing a character, which truncated every attempt. Any model whose reasoning cannot be switched off needs its budget raised, not its prompt shortened.
None of those three are model-choice problems, and none of them would have surfaced from reading model cards. They surfaced because something ran the real prompt, at the real settings, and checked the output against a rule. That is also the argument for verifying AI output rather than trusting it: the failures that matter are the ones that look fine from the outside. Ours looked fine for months.
What this does not show
The tables above are a decision record for one product, not a ranking of vendors. Specifically:
It is not a vendor ranking
Four to nine fixtures per job, mostly one pass each. These numbers say which model to point one job at, on one day, under one prompt. Reading them as a statement about which lab makes better models would be wrong in both directions.
The fixtures are ours and they are public
They live in the repository, which means anyone can read them, including us. Treat a future re-run of an unchanged fixture set as a regression check rather than fresh evidence.
Single runs lie, and we can prove it
The mode classifier runs at temperature 1.0. Three consecutive single passes ranked three different models first. Only after running every case three times did a stable answer appear, and the model that won the first pass finished third on the average. Every job with one pass per cell carries that same risk; we have only paid for repeats on the one where it bit us.
One of our labels was wrong
A test case we had marked as unambiguous was called ambiguous by all five models. Re-reading our own specification, the models were right and the label was wrong. We corrected it, disclosed it, and it is worth noting that the only reason we caught it was disagreement between models that had no way to confer.
How to run this on your own product
The whole thing is about three hundred lines and one afternoon. If your product has a model ID in a config file, this is the shape of it.
List the places a model gets chosen
Not just the obvious feature. Ours included a summarising step and a classifier that no one thought of as an AI feature at all, and both were running the oldest ID in the codebase.
Write a scorer before you write fixtures
It forces the real question: what would make this output wrong? If the only answer is that it feels worse, you do not have a benchmark, and asking a model to grade it will produce a number that hides that.
Drive the production prompt, not a copy of it
Read the prompt off the real object at the real token limit. A benchmark with its own copy of a hundred-line prompt is measuring the copy from the day someone edits one of them.
Write the selection rule down first
Then encode it, so the choice is reproducible rather than remembered. Ours is a class with five tests.
Repeat anything stochastic
If the call runs above temperature zero, one pass is a coin toss with extra steps. Three passes changed our answer on one job out of six.
Make the build fail when they drift apart
A test that re-runs the selection rule over the committed results and compares it to what ships. That is the part that stops this becoming a stale model ID again in four months, which is exactly how ours got there.
The uncomfortable version of this finding is that five of six model choices in a product built by people who think about AI reliability for a living were never checked. If you have not run this on your own stack, the base rate suggests you will find the same thing. And if you are publishing work that AI helped produce, the same logic applies one level up: run a draft through the free tools and see what a check actually turns up.
Frequently Asked Questions
How do you choose which AI model to use for a feature?
Build a fixture set for the specific job, write a scorer that computes correctness from the fixture rather than asking a model, then run every candidate through the real prompt at the real settings and change nothing but the model. Write the selection rule before you look at the tables. Ours drops anything that failed a fixture or averaged over ten seconds, takes everything within five points of the best, and picks the cheapest. Across six jobs that rule changed five of our choices.
Should you use an LLM as a judge to evaluate model outputs?
Not if you can avoid it, and for well-specified jobs you usually can. An LLM judge inherits the preferences of the model doing the judging, including a measurable preference for outputs from its own family, so a benchmark scored that way quietly encodes the confound it exists to detect. Every metric in this study is computed from the fixture and the response: verdict match against a labelled answer, whether a quoted sentence actually appears in the source, whether the counts a model stated match the array it produced.
Is the cheapest AI model usually good enough?
For extraction and classification work in 2026, often yes. On three of our six jobs every candidate scored within a couple of points of every other, which means quality contributed nothing to the decision and price decided it. That is not true for generative work: on our text editing job the best model scored about four points higher and cost 17.5 times more, which is a real trade rather than a tie.
How often should you re-benchmark the models in your product?
On any frontier release that touches a model you use, and otherwise whenever the benchmark is cheap enough to run on a schedule. Ours costs 64 cents and one command. The failure we actually hit was not running it at all: three jobs were pinned to a preview build that had been superseded twice, and nothing in the system noticed because nothing was checking. A build test that compares the shipped model against the committed evidence fixes that more reliably than a calendar reminder.
Why did running the same benchmark three times give different winners?
Because the call runs at temperature 1.0, so the output is sampled rather than fixed. On our mode classifier, three consecutive single passes over the same eight cases ranked three different models first, and the model that won the first pass came third on the three-pass average. Any benchmark of a non-zero-temperature call with one pass per cell is reporting a sample, not a measurement. Repeats are the fix, and the spread between passes is worth publishing alongside the mean.
Which models does TrueStandard use now?
The free claim checker and pre-mortem generator run Gemini 3.5 Flash Lite; the sycophancy detector, the humanizer, the report consolidator and the mode classifier run GPT-5.6 Luna. Paid runs of the free tools fan out across four models from four different labs and show where they disagree. All six single-model choices trace to a committed benchmark run, and a test fails the build if any of them stops matching what the evidence picks.
Keep reading
Is There a Most Accurate AI Model?
The honest answer is no. The ranking changes with the task, the benchmark, and the month, and even the leader still hallucinates.
Why AI Is Confidently Wrong
Models sound certain every time, even when wrong. The confident tone you trust in people is worthless here. Here is the fix.
Should You Stop Using ChatGPT?
Researchers found AI made experts measurably worse on hard tasks. Here is when to trust ChatGPT, and when it is just telling you what you want to hear.
AI Hallucination Rates in 2026: What the Data Actually Shows
A sourced reference of the 2026 hallucination-rate numbers: what each benchmark measured, which models did best and worst, and where the widely quoted figures get misread. Built to be linked and kept current as new data lands.
GPTZero's Hallucination Detector, Explained
It catches citations that don't exist. It does not, by GPTZero's own admission, check whether what you wrote is true. Here is exactly what its hallucination and source tools do, where the gap is, and what closes it.
Five of six model choices were never checked. Check your draft.
The failures that matter are the ones that look fine from the outside. Paste your draft into TrueStandard and four models from different labs check every claim in parallel, flagging the ones they cannot corroborate in about 60 seconds.
Check Your Draft →