An AI fact-checker's false positive rate is how often it flags a claim that is true, and almost nobody publishes it. Recall gets published everywhere, because recall is the flattering half: a checker that flags every sentence you give it catches 100 percent of the errors and is worthless. Precision is the half that decides whether you can use the thing, and it is the half that makes a vendor look bad.
So we ran it on our own checker. 30 claims with known answers, 15 with real literature behind them and 15 built for the benchmark with none, through the product's own pipeline at both depths. The false positive rate came back between 0 and 6.7 percent. Then we looked at the interval and at the claim set, and the honest headline stopped being the rate.
The short answer
On 30 labelled claims our checker flagged nothing true under the lenient reading and one thing under the strict reading, for a false positive rate of 0 to 6.7 percent, and caught every constructed claim. Read that with the interval attached: 1.2 to 29.8 percent. At 15 claims a side the confidence interval is wider than most of what anyone would want to conclude.
The result we would rather report is that we score well. The result worth reporting is that this test cannot tell you what our false positive rate is, and we can say exactly why.
Both depths, 30 claims, one run each
| Depth | Verdicts returned | False positive rate | Recall | Cost |
|---|---|---|---|---|
| Single | 14 verified, 1 partial, 15 challenged | 0.0% to 6.7% | 100% | $0.0140 |
| Council | 14 verified, 1 partial, 15 challenged | 0.0% to 6.7% | 100% | $0.2251 |
The range on the false positive rate is the two readings of a hedge. Our checker returns three verdicts, and a claim marked partial has not been rejected. The strict reading counts partial as a flag, the lenient reading counts it as a pass, and we publish both because publishing one lets us choose the flattering half. Wilson 95% intervals: 1.2 to 29.8 percent strict on the false positive rate, 79.6 to 100 percent on recall.
Why vendors publish recall and hide precision
A checker has two ways to be wrong. It can miss something false, or it can flag something true. The first failure is invisible to the person using it, because a claim that passes looks the same whether the checker examined it or waved it through. The second failure is the one the user feels: every flag costs a few minutes of chasing a claim that was fine.
That asymmetry decides what a vendor advertises. Recall answers what the tool caught, flagging more makes it look perfect, and the cost of doing so lands on the user, not the vendor. So recall goes on the page and precision stays in the drawer.
It also explains why detector benchmarks so often find that the vendor running them wins. A 2024 comparison of hallucination detection methods for retrieval systems, written by an engineer at the company whose method topped every dataset, is the shape of the genre. We are not exempt from that pull, which is why this page publishes the test's weaknesses in the same breath as its numbers, and why we published the model bake-off that changed five of our own six choices the same way.
The reason we can publish a false positive rate at all is that our checker is not one model deciding. It runs your text past four models from different labs, and it surfaces the places they disagree instead of averaging them away. A single model has no way to tell you it was unsure.
What we ran
30 claims with labels fixed before the run. 15 are supported, meaning real peer-reviewed literature stands behind them. 15 are unsupported, meaning we wrote them for this benchmark and no literature exists. A supported claim that comes back challenged is a false positive. An unsupported claim that comes back verified is a miss.
Each claim was submitted the way a writer would submit it, as a short draft excerpt, not a bare assertion, and run through the product's real services at both depths. Not a bespoke prompt built for the test. 60 cells, no errors, 24 cents of model spend.
The scorer's arithmetic was checked against a synthetic run of known composition before we spent anything on the real one, because a scoring bug that flatters the product is the easiest mistake to make here and the hardest to notice.
The numbers came back at the ceiling
100 percent recall. One flag on a true claim, and that flag is arguable. Every number at or near its best possible value.
Our own test kit has a rule for this: if TrueStandard wins every row, the test is wrong or the claim set is rigged. It is the first one. The 15 unsupported claims were written in-house for a citation benchmark and read as invented to any capable model. The 15 supported claims are things like handwashing reducing diarrhoeal disease in children. Nothing in the set is hard to judge, so nothing in the set can find the point where a checker starts misfiring.
The one flag lands on the claim that intermittent fasting reduces circulating C-reactive protein in adults. Our label says supported. The literature is genuinely mixed. Three of four models hedged instead of passing it, which is arguably the right answer and a sign that our label is the weak part of that row, not the checker.
This is the same finding our generation benchmark hit from the other direction. Holding the model, the prompt and the scorer still and changing only the question set moved a fabrication rate from 10.9 to 30.9 percent, which is why hallucination benchmarks disagree. Difficulty dominates. An easy corpus produces a flattering number about the corpus and tells you very little about the product.
Four models changed nothing, and split anyway
Both depths returned the identical verdict on all 30 claims. Council costs 16.1 times what Single costs, 22.5 cents against 1.4, and on this corpus it moved no verdict at all.
That is a real result and it comes with a real boundary. It holds for claims this easy. It says nothing about the claims where a checker actually struggles, and quoting it as evidence that four models add nothing would be exactly the overreach this page is arguing against.
Underneath the identical verdicts, the models were not agreeing. Six of the 30 claims came back split, every one of them three to one.
The six splits
| Claim | Label | Final verdict | The dissenter |
|---|---|---|---|
| S02 | supported | partial | Gemini said verified, three said partial |
| S09 | supported | verified | Claude said partial |
| S12 | supported | verified | Grok said partial |
| U09 | unsupported | challenged | Claude said partial |
| U14 | unsupported | challenged | Gemini said partial |
| U15 | unsupported | challenged | Grok said partial |
A single model returns one verdict and no way to know it was a close call. On a fifth of these claims the panel was split, and a reader who saw only the majority answer would never learn which fifth. That split is the thing worth paying for, and it is what a Council run puts on the screen.
Every disagreement was a hedge
In all six splits the minority model returned partial where the majority committed. Not once did a model say verified while another said challenged. The panel never disagreed about direction, only about how sure to be.
That is a more useful property than unanimity would be. A checker whose models split on direction would be unusable, because the majority verdict would be a coin weighted by whichever vendors happened to be in the panel. A checker whose models agree on direction and vary on confidence is telling you something real: this claim sits near the edge of what can be settled from the text alone.
No model was the systematic sceptic either. The dissenter was Gemini twice, Claude twice, Grok twice, and GPT never. At six splits that is far too little to call a house style, and we are naming the counts and stopping there.
What a harder test needs
The false positive rate lives in claims that are true and do not look it. Surprising findings, precise statistics, niche results, things that read like something a careless writer invented and happen to be documented. Our supported claims are none of those, which is why nothing misfired.
The unsupported side needs the mirror. Plausible constructions that do not announce themselves, in place of sentences built to be caught. Both halves of the current set announce themselves, and a test where both strata are obvious measures whether a model can read.
We built that corpus the same day and ran it, so the number is in rather than promised. 12 mirror pairs, each true claim asserting a curated answer from a public third-party benchmark and each false one the same sentence with a single value swapped. On that set the false-positive rate is 25 to 75 percent depending on how a hedge is counted, against 0 to 6.7 percent here, and single-model accuracy on a balanced set falls to a coin flip.
Two things follow. The rate on this page is a fact about an easy corpus and should not be quoted as our false-positive rate. And the finding above, that four models changed no verdict, reverses on hard claims: Council corrected three misses out of 24 and introduced one false positive, buying recall at a small cost in precision. A rate that only ever appears after it looks good is not a measurement, which is why the flattering one is still on this page.
The habit is the product. Run the five questions on any accuracy claim you are handed, ours included: what corpus, grounded or ungrounded, how many runs, what counted as a failure, and who scored it. This page fails the first one on purpose, and says so.
Frequently Asked Questions
What is a false positive rate for an AI fact-checker?
It is how often the checker flags a claim that is actually true. A checker with a high false positive rate sends you chasing sources for claims that were already fine, which costs more time than it saves. It is the counterpart to recall, which measures how many false claims the checker caught, and a tool that flags everything scores perfectly on recall while being useless.
What is TrueStandard's false positive rate?
On 30 labelled claims run on 30 August 2026, between 0 and 6.7 percent depending on whether a hedged verdict counts as a flag, with a 95 percent confidence interval of 1.2 to 29.8 percent on the stricter reading. We do not think that is a trustworthy estimate, because the claim set was too easy to stress a checker, and we say so on the page instead of quoting the number alone.
Why publish a benchmark that says your own test was flawed?
Because a rate that only appears when it looks good is not a measurement. Our test kit carries a rule that if we win every row, the test is wrong, not the product perfect, and this run triggered it. Publishing the weakness first is what makes the harder run that follows worth reading.
Is running four models better than one for fact-checking?
On this set it changed no verdict while costing 16 times more, so on claims this easy one model is enough. It still split three to one on six of the 30 claims, and a single model gives you a verdict with no signal that it was a close call. The value of the panel is knowing which claims were contested, not getting a different answer.
What is the difference between recall and precision in fact-checking?
Recall is the share of false claims the checker caught. Precision is the share of its flags that were correct. They pull against each other: flagging more raises recall and lowers precision. A vendor quoting only recall is quoting the number that improves when the tool gets more aggressive, which is the direction that costs you time.
How do I test an AI fact-checker myself?
Build a set of claims where you already know the answer, half true and half false, and submit them the way you would submit real work. Count both errors separately: true claims it flagged, and false claims it passed. Make the true half hard, with surprising and precise claims, because that is where a checker misfires and where an easy set will tell you nothing.
What does a partial verdict from a fact-checker mean?
It means the claim was neither confirmed nor rejected on the evidence available. In our run every disagreement between models was a hedge of this kind rather than a reversal, so no model ever called a claim true while another called it false. A hedge is a signal that the claim sits at the edge of what the text alone can settle, and it is worth a human look.
Keep reading
Why AI Hallucinations Are Structural
DELEGATE 52, GPT-5.5, and a Purdue impossibility proof. Three April 2026 results that move 'hallucinations are structural' from take to documented fact.
When AI Cites Studies That Don't Exist
AI does not just get facts wrong. It invents whole sources, cases, studies, DOIs, and cites them with the same confidence it uses for real ones. Here is why it happens, the disasters it has already caused, and how to catch a fabricated citation before your name is on it.
AI Fake-Citation Disasters: A 2026 Reference
A running catalogue of documented cases where AI fabricated citations, across law, academia, and media. Every entry is sourced. It is here to be linked, cited, and updated as new cases surface.
How to Check If AI Citations Are Fake
Four checks catch a fabricated reference before your readers do. One of them is new: in 2026, a DOI that resolves no longer means the citation is real.
GPTZero's Hallucination Detector, Explained
It catches citations that don't exist. It does not, by GPTZero's own admission, check whether what you wrote is true. Here is exactly what its hallucination and source tools do, where the gap is, and what closes it.
See where four models disagree on your draft
A single model gives you one verdict and no way to know it was close. TrueStandard runs your text past four models from different labs and shows you every claim they split on.
Start Verifying →