An AI sycophancy benchmark measures whether a model holds its answer when you push back. We ran one across nine models and 1,620 calls, then found something in our own results that we had not gone looking for. The two models with the lowest sycophancy rates in the field were the two we could barely score. They had returned a readable verdict on 56% and 58% of their cells. Everyone else managed 87% to 100%.
A cell with no answer never shows the model changing its answer. So it leaves the denominator and the rate improves, not because the model resisted the push but because it never met it. Every reliability metric shaped as a ratio has this property, which explains a good deal of why published sycophancy scores put the same model in opposite places.
The short answer
A model that stops answering scores well on almost any reliability benchmark, because most of them are ratios and an unanswered item quietly leaves the bottom of the fraction. In our August 2026 run, MiniMax M3 and DeepSeek V4 Pro posted sycophancy rates of 2.0% and 1.9%, the two lowest of nine models, on 56% and 58% of their cells. We publish both numbers and rank neither of them. The full table, with intervals, is on the AI sycophancy benchmark.
A better scorer does not fix this. A second column does. A benchmark that prints a rate without printing how much of the test each model finished gives you one number when the job takes two, and the missing one is the one that tells you whether to trust the first.
What we found in our own data
Nine models judged the same thirty frozen claims as supported, unsupported or uncertain. Then we told each one its verdict was wrong, in the words a real user would use, and asked again. Sycophancy is the share of claims where the second verdict moved toward the push. Three runs, 90 cells per model per arm, no errors.
| Model | Cells we could score | Sycophancy rate |
|---|---|---|
| Gemini 3.7 Flash | 100% | 0.0% |
| Grok 4.3 | 100% | 3.3% |
| GPT-5.6 Luna | 100% | 6.7% |
| DeepSeek V4 Flash | 87% | 11.5% |
| DeepSeek V4 Pro | 58% | 1.9% |
| MiniMax M3 | 56% | 2.0% |
Read the last two rows against the rest. On the numbers alone they are the most principled models in the field. On the second column they are the two we know least about. Their intervals are also the widest, because an interval widens as the sample shrinks, which is the statistics quietly telling you the same thing the answer rate does.
The other three models are in the full board. Prices, intervals and the method are there too.
Why sycophancy benchmarks contradict each other
This is not only our problem. Published sycophancy benchmarks disagree about the same models in opposite directions, and most write-ups pin that on prompt wording. Measurement completeness explains part of it.
| Benchmark | What it counts | Where a quiet model helps itself |
|---|---|---|
| Preference-revealing evals | Whether stating a preference changes the answer | A refusal counts as no change |
| Multi-turn conformity tests | How many turns until the model flips | A model that ends the turn early never flips |
| Ours, v7 | Verdict movement after one push | An unreadable verdict leaves the denominator |
One published evaluation ranks Gemini the most sycophantic model it tested. Another ranks it the least. Both teams ran their tests competently. When two careful measurements of the same behaviour in the same model point in opposite directions, the definition is doing more work than the data, and the first thing to check is what each one did with the items nobody answered.
We hit the same wall from the other side when we measured fabrication. A model that returns an empty response offers zero citations, and zero citations means zero fabricated ones, so it posts a perfect score. That is why our fabrication board publishes an answer rate beside every rate, and why the model with the lowest fabrication rate there finishes third on what a reader actually receives.
The denominator is doing the work
Every rate in this field is a fraction. Arguments almost always circle the numerator. The denominator moves the number more.
The model does not answer
It returns an empty string, or prose where a verdict was required, or it spends its whole token budget reasoning and stops before writing anything. None of these is an API error. The call succeeds and returns 200.
The scorer cannot grade it
A cell with no readable verdict cannot be compared against the pressured verdict, so it is dropped. Dropping it is correct: counting it as held ground would be worse. But the drop is silent.
The rate improves
Fewer chances to be caught moving means a lower rate. The model is rewarded for the exact behaviour that makes it less useful, and nothing in the headline number says so.
The uncomfortable part is that every step here is defensible in isolation. The model is allowed to decline. The scorer is right to exclude what it cannot read. The rate is computed correctly. The failure is emergent, it only appears when you rank, and ranking is the one thing everybody does with these numbers.
How to read any reliability benchmark
Four questions, in this order. They take about a minute and they will disqualify most leaderboards you meet.
What share of the test did each model complete?
If this is not published, the ranking is unreadable and you should stop. A model that answered 56% of the items is not comparable to one that answered 100%, whatever the rates say.
What happened to the items nobody answered?
Dropped, counted as passes, or counted as failures. All three are defensible and all three produce different orderings from identical data.
Are intervals published, and do they overlap?
Most published gaps between adjacent models are smaller than their intervals. In our own field, only two of eight pairwise comparisons separate at 95%, so we say so on the page rather than printing a clean list of nine.
How many runs?
One run of this test has told us the opposite of the next two, on identical inputs, on the same afternoon. A single-run benchmark is a sample of one dressed as a measurement.
None of this makes benchmarks useless. It makes a single number from one useless. The same discipline is why we run every claim past several models rather than trusting one, and why asking a model to check itself does not work: one measurement of one thing, however careful, is still one point of view.
What we changed
We could have published nine ranked rows. The numbers were computed correctly and nobody would have queried them.
Instead the board ranks seven models and lists two below a rule, with their scores shown and their positions withheld. The threshold is 85% of cells scorable, which is not a principled constant, it is the gap that appeared in our own data between 87% and 58% with nothing in between. We would rather state an arbitrary line and show where it falls than quietly drop two models or quietly rank them.
A benchmark that hides what it could not measure is not reporting a result. It is reporting the subset of reality that happened to be easy to score, and calling the leftovers a ranking.
Frequently asked questions
What is an AI sycophancy benchmark?
A test of whether a model holds its answer when the user disagrees with it. The standard shape is two turns: ask the model to judge something, record the answer, then assert the opposite and ask again. The sycophancy rate is the share of items where the second answer moves toward what you asserted. It measures whether a model can be talked out of a position, not whether the position was correct.
Why would a model that answers less score better?
Because these metrics are ratios. An item the model did not answer cannot be observed changing, so most scorers exclude it from the denominator rather than count it as a pass or a failure. Excluding it is the least-wrong option, but it means every unanswered item is one fewer chance to be caught, and a model that answers half the test gets half the exposure. The rate falls without the behaviour improving.
Which AI model is least sycophantic?
In our August 2026 measurement, Gemini 3.7 Flash moved on none of its 90 scored cells and Qwen3.8 Max on none of its 85. Their 95% intervals reach 4.1% and 4.3%, so the defensible claim is that neither moved on anything we tested. Three further models cluster between 3.3% and 4.4% and cannot be separated from one another on this sample.
Why do published sycophancy benchmarks disagree about the same model?
Partly because they define the behaviour differently, and partly because they handle unanswered items differently. One evaluation ranks Gemini the most sycophantic model it tested while another ranks it the least. Before comparing two sycophancy scores, check that both counted the same thing and both did the same thing with the items nobody answered.
Does a low sycophancy score mean a model is accurate?
No. Sycophancy measures whether a model holds a position under pressure, not whether the position was right. A model that is confidently wrong and completely immovable scores perfectly. This is why sycophancy and fabrication have to be published as two numbers and never averaged into one reliability score.
Does the same problem affect hallucination benchmarks?
Yes, and more sharply. An empty response contains no citations, so it contains no fabricated ones, and it posts a fabrication rate of zero. A model that cannot answer will top a single-column fabrication leaderboard. Any board worth reading publishes an answer rate in the next column.
Where should the cutoff for ranking a model sit?
There is no principled constant. We used 85% of cells scorable because our own data had a gap there, between 87% and 58%, with nothing in between. What matters is that the threshold is stated, that models below it are shown rather than deleted, and that the reader can move the line themselves because the underlying numbers are published.
How many runs does a reliability benchmark need?
More than one, and the reason is empirical rather than theoretical. Running the same test three times on the same afternoon with the same inputs has produced orderings that contradicted each other. We publish pooled rates across three runs plus the per-run range, so a reader can see how much of any gap is real and how much is noise.
Keep reading
Is There a Most Accurate AI Model?
The honest answer is no. The ranking changes with the task, the benchmark, and the month, and even the leader still hallucinates.
Why AI Is Confidently Wrong
Models sound certain every time, even when wrong. The confident tone you trust in people is worthless here. Here is the fix.
Should You Stop Using ChatGPT?
Researchers found AI made experts measurably worse on hard tasks. Here is when to trust ChatGPT, and when it is just telling you what you want to hear.
AI Hallucination Rates in 2026: What the Data Actually Shows
A sourced reference of the 2026 hallucination-rate numbers: what each benchmark measured, which models did best and worst, and where the widely quoted figures get misread. Built to be linked and kept current as new data lands.
AI Detector vs Fact Checker
One asks who wrote this. The other asks is this true. Before you publish, only one of those questions protects your reputation — and most teams are watching the wrong one.
One number, one model, one run. Pick any two and you still have a guess.
TrueStandard runs your draft past four frontier models from different vendors at once and shows you where they disagree. A claim that survives only because one model went quiet does not survive here.
Start Checking →