An AI fact-checker's false positive rate is how often it flags a claim that is true. Almost nobody publishes it. Recall gets published everywhere, because recall is the flattering half. A checker that flags every sentence you give it catches 100 percent of the errors and is worthless. Precision is the half that decides whether you can use the thing, and the half that makes a vendor look bad.
So we ran it on our own checker. 30 claims with known answers: 15 with real literature behind them, and 15 built for the benchmark with none. All of them went through the product's own pipeline, at both depths, and the false positive rate came back between 0 and 6.7 percent. Then we looked at the interval and at the claim set. The headline stopped being the rate.
The short answer
On 30 labelled claims our checker flagged nothing true under the lenient reading, and one thing under the strict reading. That is a false positive rate of 0 to 6.7 percent. It caught every constructed claim, and with the interval attached the range is 1.2 to 29.8 percent. At 15 claims a side, the confidence interval is wider than most of what anyone would want to conclude.
The result we would rather report is that we score well. The result worth reporting is that this test cannot tell you what our false positive rate is.
Both depths, 30 claims, one run each
| Depth | Verdicts returned | False positive rate | Recall | Cost |
|---|---|---|---|---|
| Single | 14 verified, 1 partial, 15 challenged | 0.0% to 6.7% | 100% | $0.0140 |
| Council | 14 verified, 1 partial, 15 challenged | 0.0% to 6.7% | 100% | $0.2251 |
The range on the false positive rate is the two readings of a hedge. Our checker returns three verdicts, and a claim marked partial has not been rejected. The strict reading counts partial as a flag, and the lenient reading counts it as a pass. We publish both, because publishing one lets us choose the flattering half. Wilson 95% intervals: 1.2 to 29.8 percent strict on the false positive rate, 79.6 to 100 percent on recall.
Why vendors publish recall and hide precision
A checker has two ways to be wrong: it can miss something false, or it can flag something true. The first failure is invisible to the person using it. A claim that passes looks the same whether the checker examined it or waved it through. The second failure is the one the user feels, because each flag costs a few minutes of chasing a claim that was fine.
That asymmetry decides what a vendor advertises, and recall answers what the tool caught. Flagging more makes it look perfect, and the cost of that lands on the user. So recall goes on the page and precision stays in the drawer.
It also explains why detector benchmarks often find that the vendor running them wins. Take a 2024 comparison of hallucination detection methods for retrieval systems, written by an engineer at the company whose method topped every dataset. That is the shape of the genre. We are not exempt from that pull, so this page publishes the test's weaknesses with its numbers. It is also why we published the model bake-off that changed five of our own six choices the same way.
The reason we can publish a false positive rate is that our checker is not one model deciding. It runs your text past four models from different labs, and surfaces the places they disagree instead of averaging them away. A single model has no way to tell you it was unsure.
What we ran
30 claims with labels fixed before the run. 15 are supported, meaning real peer-reviewed literature stands behind them. 15 are unsupported, meaning we wrote them for this benchmark and no literature exists. A supported claim that comes back challenged is a false positive, and an unsupported claim that comes back verified is a miss.
Each claim was submitted the way a writer would submit it, as a short draft excerpt. Each one ran through the product's real services, at both depths. 60 cells, no errors, 24 cents of model spend.
The scorer's arithmetic was checked against a synthetic run of known composition, before we spent anything on the real one. A scoring bug that flatters the product is the easiest mistake to make here. It is also the hardest to notice.
The numbers came back at the ceiling
100 percent recall. One flag on a true claim, and that flag is arguable.
Our test kit has a rule for this. If TrueStandard wins every row, the test is wrong or the claim set is rigged. It is the first one. The 15 unsupported claims were written in-house for a citation benchmark, and they read as invented to any capable model. The 15 supported claims are things like handwashing reducing diarrhoeal disease in children. Nothing in the set is hard to judge, so nothing in the set can find the point where a checker starts misfiring.
The one flag lands on the claim that intermittent fasting reduces circulating C-reactive protein in adults. Our label says supported; the literature is mixed. Three of four models hedged instead of passing it. That is arguably the right answer, and a sign that our label is the weak part of that row.
This is the same finding our generation benchmark hit from the other direction. We held the model, the prompt and the scorer still, and changing only the question set moved a fabrication rate from 10.9 to 30.9 percent. That is why hallucination benchmarks disagree. An easy corpus produces a flattering number about the corpus, and tells you little about the product.
Four models changed nothing, and split anyway
Both depths returned the identical verdict on all 30 claims. Council costs 16.1 times what Single costs, 22.5 cents against 1.4. On this corpus it moved no verdict.
It holds for claims this easy. It says nothing about the claims where a checker struggles, and quoting it as evidence that four models add nothing would be overreach.
Underneath the identical verdicts, the models were not agreeing. Six of the 30 claims came back split, every one of them three to one.
The six splits
| Claim | Label | Final verdict | The dissenter |
|---|---|---|---|
| S02 | supported | partial | Gemini said verified, three said partial |
| S09 | supported | verified | Claude said partial |
| S12 | supported | verified | Grok said partial |
| U09 | unsupported | challenged | Claude said partial |
| U14 | unsupported | challenged | Gemini said partial |
| U15 | unsupported | challenged | Grok said partial |
A single model returns one verdict and no way to know it was a close call. On a fifth of these claims the panel was split. A reader who saw only the majority answer would never learn which fifth. That split is the thing worth paying for, and it is what a Council run puts on the screen.
Every disagreement was a hedge
In all six splits the minority model returned partial where the majority committed. The panel never disagreed about direction, only about how sure to be.
That is a more useful property than unanimity would be. A checker whose models split on direction would be unusable, because the majority verdict would be a coin weighted by whichever vendors happened to be in the panel. A checker whose models agree on direction and vary on confidence tells you something else. This claim sits near the edge of what can be settled from the text alone.
No model was the systematic sceptic either. The dissenter was Gemini twice, Claude twice, Grok twice, and GPT never. At six splits that is too little to call a house style.
What a harder test needs
The false positive rate lives in claims that are true and do not look it: surprising findings, precise statistics, niche results. Things that read like something a careless writer invented, and happen to be documented. Our supported claims are none of those, which is why nothing misfired.
The unsupported side needs the mirror: plausible constructions that do not announce themselves. Both halves of the current set announce themselves, and a test where both strata are obvious measures whether a model can read.
We built that corpus the same day and ran it, so the number is in. 12 mirror pairs: each true claim asserts a curated answer from a public third-party benchmark, and each false one is the same sentence with a single value swapped. On that set the false-positive rate is 25 to 75 percent, depending on how a hedge is counted. Here it was 0 to 6.7 percent. And single-model accuracy on a balanced set falls to a coin flip.
The rate on this page is a fact about an easy corpus, and it should not be quoted as our false-positive rate. And the finding above, that four models changed no verdict, reverses on hard claims. Council corrected three misses out of 24 and introduced one false positive, buying recall at a small cost in precision.
Run the five questions on any accuracy claim you are handed: what corpus, grounded or ungrounded, how many runs, what counted as a failure, and who scored it.
Frequently Asked Questions
What is a false positive rate for an AI fact-checker?
It is how often the checker flags a claim that is true. A checker with a high false positive rate sends you chasing sources for claims that were already fine, which costs more time than it saves. It is the counterpart to recall, which measures how many false claims the checker caught. A tool that flags everything scores perfectly on recall while being useless.
What is TrueStandard's false positive rate?
On 30 labelled claims run on 30 August 2026, between 0 and 6.7 percent. The range depends on whether a hedged verdict counts as a flag. The 95 percent confidence interval is 1.2 to 29.8 percent on the stricter reading. We do not think that is a trustworthy estimate, because the claim set was too easy to stress a checker.
Why publish a benchmark that says your own test was flawed?
Because a rate that only appears when it looks good is not a measurement. Our test kit carries a rule that if we win every row, the test is wrong, and this run triggered it.
Is running four models better than one for fact-checking?
On this set it changed no verdict while costing 16 times more, so on claims this easy, one model is enough. It still split three to one on six of the 30 claims. A single model gives you a verdict with no signal that it was a close call. The value of the panel is knowing which claims were contested.
What is the difference between recall and precision in fact-checking?
Recall is the share of false claims the checker caught. Precision is the share of its flags that were correct. They pull against each other: flagging more raises recall and lowers precision. A vendor quoting only recall is quoting the number that improves when the tool gets more aggressive. That is the direction that costs you time.
How do I test an AI fact-checker myself?
Build a set of claims where you already know the answer, half true and half false. Submit them the way you would submit real work. Count both errors separately: true claims it flagged, and false claims it passed. Make the true half hard, with surprising and precise claims. That is where a checker misfires, and where an easy set will tell you nothing.
What does a partial verdict from a fact-checker mean?
It means the claim was neither confirmed nor rejected on the evidence available. In our run every disagreement between models was a hedge of this kind. No model ever called a claim true while another called it false. A hedge is a signal that the claim sits at the edge of what the text alone can settle. It is worth a human look.
Keep reading
Why AI Hallucinations Are Structural
DELEGATE 52, GPT-5.5, and a Purdue impossibility proof. Three April 2026 results that move 'hallucinations are structural' from take to documented fact.
Can Jev Fact-Check Claims? It Declined 12 of 13
We handed TypeSafe's decision model thirteen factual claims with nothing to check them against. It refused to guess on twelve of them, which is the best thing we can say about it.
We Put Jev in Our Pipeline and Saved Nothing
The per-judgment price said we would cut the bill by a quarter. Measured against the real pipeline it saved nothing at all, and the reason is more useful than the saving would have been.
When AI Cites Studies That Don't Exist
AI does not just get facts wrong. It invents whole sources: cases, studies, DOIs. Then it cites them with the same confidence it uses for real ones. Here is why it happens, and the disasters it has already caused. And here is how to catch a fake citation before your name is on it.
GPTZero Hallucination Detector, Explained
It catches citations that don't exist. By GPTZero's own admission, it does not check whether what you wrote is true. Here is exactly what its hallucination and source tools do. Here is where the gap is, and what closes it.
See where four models disagree on your draft
A single model gives you one verdict and no way to know it was close. TrueStandard runs your text past four models from different labs, and shows you each claim they split on.
TrueStandard is for writers who publish AI-assisted drafts. Paid plans check every claim and citation across four AI models before you hit publish.
Free: one check a day, no signup. Pro is $20 a month for 250 drafts.
Start Verifying →