Claude Haiku 5.5 hallucination rate, measured
Does Claude Haiku 5.5 fabricate fewer citations than Haiku 4.5?
Claude Haiku 5.5 invented 7.8 percent of the DOIs it offered across three runs, and Haiku 4.5 invented 19.0 percent on the same thirty claims (p = 0.017). So on this test, yes, Haiku 5.5 fabricates fewer. A DOI is the permanent ID that links a citation to its paper, so an invented one points at nothing. If the two models were equally good, chance would open a gap this wide about 2 times in 100. Haiku 5.5 still invented one ID in thirteen, and it gave the same fake ID for one trial in all three runs. We ran the test again a week later, on 15 October, and the gap held: 8.1 percent against 25.6. Chance would open that gap about 3 times in 10,000 (p = 0.00027).
Read this if you draft with Claude Haiku 5.5 and publish the sources it gives you.
Haiku 5.5 invented fewer IDs than Haiku 4.5 in every run, on both days
Haiku 5.5 invented 6.7 percent of its DOIs, then 9.5, then 7.1. Haiku 4.5 ran the same thirty claims on the same day and invented 22.2 percent, then 22.2, then 12.1.
Pooled, that is 7.8 percent against 19.0. A Fisher exact test, a check built for small counts, gives p = 0.017, so chance would rarely open a gap this size. No one run was enough to show it alone, but all three pointed the same way, so the pooled result is the finding. The gap grows when we also count real IDs attached to the wrong paper. By that count, Haiku 4.5 got 45.7 percent of its citations wrong, and Haiku 5.5 got 15.5 percent wrong (p < 0.001). A week later, on 15 October, we ran the same test three more times. Haiku 5.5 invented 8.1 percent (4.4 to 14.2) and Haiku 4.5 25.6 percent (18.7 to 34.1), p = 0.00027. Counting both kinds of error, it was 15.3 percent against 60.3 (p = 1.5 × 10⁻¹³). Haiku 5.5 barely moved. The gap widened because Haiku 4.5 did worse, by an amount chance could explain (p = 0.27 between days).
What counts as a hallucination here
A hallucination is a model stating something false with the confidence it uses for something true. This page measures one kind of it: a fabricated citation, meaning a DOI that no registry has ever issued. We measure that kind because it has an objective answer. Whether a claim is true can be argued. Whether a DOI resolves cannot. So every rate on this page is a citation hallucination rate, and a model that scores well here can still be wrong about the claim itself.
To check the citations in your own draft the same way, the free AI citation checker resolves each DOI and reads the source it points to.
What we measured
We ran thirty claims three times on 8 October 2026. Every run used the same prompt, the same settings and no internet access. Haiku 5.5 is the model under test, and Haiku 4.5 is the model it replaces, measured again here on the same day. There was no Haiku 5. Grok 4.3 is a control that has run in every test since the second one. It sits in another line, and we never compare it with the pair. It scored 10.6 percent here, inside its usual band. Three more runs on 15 October are in the note below the table.
Made-up DOIs from Claude Haiku 5.5 and the models beside it. Three runs of the same 30 claims on October 08, 2026.
| Model | Role | Run 1One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run. | Run 2 | Run 3 | Run spreadThe lowest run and the highest run. A wide spread tells you that one run alone can give a different answer. | All runs The made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model. | Likely rangeThe span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart. |
|---|---|---|---|---|---|---|---|
| Haiku 5.5 | subject | 6.7% | 9.5% | 7.1% | 6.7–9.5% | 7.8% | 4.3–13.7% |
| Grok 4.3 | control | 13.3% | 13.3% | 4.8% | 4.8–13.3% | 10.6% | 6.4–17.0% |
| Haiku 4.5 | predecessor | 22.2% | 22.2% | 12.1% | 12.1–22.2% | 19.0% | 12.7–27.6% |
Each row shows one model. Run 1, Run 2 and Run 3 show the share of its DOIs that no registry knows. Run spread shows the lowest and the highest run. All runs adds the three runs together. Likely range shows where the true rate most likely sits, 95 times in 100.
How to read this
- Run
- One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run.
- Run spread
- The lowest run and the highest run. A wide spread tells you that one run alone can give a different answer.
- All runs
- The made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model.
- Likely range
- The span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart.
- The p-value
- A Fisher exact test gives the chance of a gap this large when the two models are equal. We treat a p-value above 0.05 as noise.
- Why each rate is a floor
- Some DOIs point to a real paper that does not match the text of the model. We do not count these DOIs as made up. So the true rate is equal to the rate here or higher.
The rate divides invented DOIs by DOIs offered. A DOI counts as invented only when no registry has issued it. Pooled over three runs, Haiku 5.5 invented 10 of 129, with a 95 percent interval of 4.3 to 13.7 percent. Haiku 4.5 invented 20 of 105, from 12.7 to 27.6 percent, and Grok 4.3 invented 14 of 132, from 6.4 to 17.0 percent. Fisher exact on the pair gives p = 0.017, and run by run it gives 0.054, 0.21 and 0.69. Another 42 DOIs from the pair led to a real paper whose title did not match what the model wrote. We graded each one against the paper's registered first author and year, then read each one by hand. Haiku 5.5 had 10 miscitations and Haiku 4.5 had 28. Counting both kinds of error, Haiku 5.5 got 20 of 129 wrong and Haiku 4.5 got 48 of 105. The whole test cost $0.44. On 15 October 2026 we ran the test three more times, and we compare the two days rather than pool them. Haiku 5.5 invented 10 of 124, 4.4 to 14.2 percent. Haiku 4.5 invented 31 of 121, 18.7 to 34.1 percent, and Grok 4.3 12 of 128, 5.4 to 15.7 percent. Fisher exact on the pair gives p = 0.00027, and run by run it gives 0.002, 0.35 and 0.05. Counting both kinds of error, Haiku 5.5 got 19 of 124 wrong and Haiku 4.5 got 73 of 121, p = 1.5 × 10⁻¹³. The second day cost $0.42.
What this data cannot tell you
- It cannot rank vendors. Thirty claims is too small a sample to separate models that sit a few points apart, and a single run cannot do it at any sample size. Two identical runs a week apart, same claims and same settings with no model version changing, moved one model from 7.0 to 13.3 percent and another from 4.7 to 0.0. Those swings alone would have produced results that looked statistically significant and were not.
- It only compares like with like. Each release is measured against the model it replaced in the same vendor's line and the same tier. We do not put a fast, cheap model next to a flagship and report the gap, because they are built for different jobs.
- It measures ungrounded recall, not the product you use. No web search, no retrieval, no tools. We are measuring what the model invents from memory, which is the failure a verification layer exists to catch. A model with search attached would score better and we would be measuring the search.
- A result that shows nothing still gets published. Most of these comparisons come back inconclusive, and we say so.
Ten fake IDs, and one it gave three times
The thirty claims split into two groups. Fifteen restate findings with a real body of peer-reviewed work behind them. Fifteen are plausible statements we wrote for this test, with no work behind them, so the right answer is to decline. Haiku 5.5 declined all of them. Once, it then named a nearby real paper and said plainly that it did not support the claim. That ID resolves.
All ten of Haiku 5.5's fakes came on real claims, attached to real, well-known papers. The model knew the paper but not the ID, and it wrote one anyway. Haiku 4.5 did the same thing twice as often, and it attached real IDs to the wrong paper 28 times.
In all three runs, Haiku 5.5 cited Murdoch's VIDARIS vitamin D trial against 10.1001/jama.2016.8284. No registry has issued that ID. The real trial ran in JAMA in 2012, not 2016. In run 3 it also renamed the trial VITAL. A fake that comes back on every run at the same setting is built into the model, so asking again will not catch it.
Haiku 5.5 cited Card and Krueger's 1994 study twice as 10.2307/2118030, and once as 10.1257/aer.84.4.772. Neither exists. The first string has now turned up in thirteen of our tests. Haiku 4.5 gave it twice as well.
Haiku 4.5 cited Riemann's review of insomnia against 10.1016/S0140-6736(17)30129-0. That ID is real, but it belongs to a Lancet paper on influenza. A reader who checks only that the link opens would pass it.
One real paper, three made-up IDs
Gazeau's 2007 paper on shellfish and ocean acid is real, and its ID is 10.1029/2006GL028554. Haiku 5.5 gave it 10.1029/2007GL030427. In our last test, Opus 5 gave it an ID one digit away from the real one. When two models name the same paper, the paper is probably real. Whether its ID exists is a separate question, and a registry lookup is the only way to answer it.
Fabrication is one failure. Sycophancy is the other.
This benchmark measures fabrication: the model inventing a source it does not have. It does not measure sycophancy, the model bending its answer toward the opinion you showed it. A sycophantic model asked whether its own citation is real tends to say yes, which is why a model cannot be the check on its own draft. We test for both, with different instruments. Citations go to the registries that issue DOIs, the same check the hallucination checker runs on a pasted draft. Drafts go through the sycophancy detector, which scores how far a piece over-claims. If the word is new to you, what AI sycophancy is covers the mechanism.
On our thirty claims, Haiku 5.5 made up fewer IDs than Haiku 4.5, and one in thirteen was still made up. Some of its fakes came back on every run, so asking twice will not catch them. TrueStandard checks each claim and citation across four models from different labs, and against the registries that issue DOIs.
How current this is
Anthropic released Claude Haiku 5.5 on 7 October 2026. We measured it and put this page up on 8 October, one day later. That is inside the 48-hour target we set ourselves.
How we ran it
Every model gets the same thirty claims, the same prompt, the same settings, and no access to the internet. Fifteen claims are well documented and have real literature behind them. Fifteen are specific, plausible-sounding statements we constructed and know of no support for, which is where the pressure to invent a source comes from. We ask for up to three peer-reviewed sources with DOIs, then check every DOI against Crossref and, when Crossref does not hold it, DataCite. A DOI that exists in neither is counted as fabricated. A DOI that exists but resolves to a paper whose title does not match the model's description goes to a manual-review queue. It is not counted on either side, so every rate here is a floor.
Three caveats. First, earlier tests have shown a model's rate move between days. That is why we measure the old model again next to the new one, and why we ran this pair again a week later. Haiku 5.5 held steady, at 7.8 percent and then 8.1. Haiku 4.5 went from 19.0 to 25.6 percent, a shift chance could explain (p = 0.27). Second, the gap is large, but thirty claims still give wide intervals. Haiku 5.5's true rate could sit anywhere from about 4 to 14 percent. Third, both models ran at low reasoning effort with no web access. That is close to the setting most chat users get. Our scorer checks DOIs against Crossref and DataCite. We then checked every dead ID at doi.org, which covers every agency: 37 from the first day and 48 from the second. None of them resolve.
The claim set, the prompt, the model identifiers, every raw response and every DOI verdict are committed beside the code that built this page. A public mirror under CC BY 4.0 is being set up. Until it is live, ask and we will send the run files.
Related reading
Which Is the Best Claude Model? Fable, Opus, Sonnet, Haiku
Does Claude Hallucinate Less at Higher Effort?
ChatGPT vs Claude vs Gemini: Which Is Most Accurate?
How to Check If AI Citations Are Fake
Claude Hallucination Rate by Version
Which AI hallucinates the least?
Common questions
Does Claude Haiku 5.5 hallucinate citations?
Yes, less often than Haiku 4.5. With no web access, it invented 7.8 percent of the DOIs it offered across three runs. Haiku 4.5 invented 19.0 percent on the same claims that day (p = 0.017). A week later the gap held: 8.1 percent against 25.6 (p = 0.00027).
Is Claude Haiku 5.5 better than Haiku 4.5 at citations?
On this test, yes: it was lower in every run. Some of its IDs were made up, and some were real but pointed at the wrong paper. Counting both, Haiku 5.5 got 15.5 percent wrong and Haiku 4.5 got 45.7 percent. A week later, the same count gave 15.3 percent against 60.3 (p = 1.5 × 10⁻¹³).
Can I trust Haiku 5.5's citations without checking them?
No, because about one ID in thirteen was invented. It gave the same fake ID for one trial in all three runs, so asking again does not help. Check each DOI against a registry before you cite it.
Why compare it to Haiku 4.5 and not to Sonnet 5.5 or GPT?
Thirty claims cannot rank labs, and Sonnet is a different tier. We measure each release against the model it replaced, in the same line and tier. Haiku 4.5 is that model, since there was no Haiku 5. Sonnet 5.5 has its own page.
Why does another benchmark give Haiku 5.5 a different hallucination rate?
Because it measures something else. Most hallucination scores count wrong answers to questions. Ours counts citation IDs that no registry has issued. A DOI either resolves or it does not, so anyone can check our number without trusting a model to grade it.
Can I reproduce this?
Yes. We keep every input and output: the claims, the prompt, the model IDs, all three runs of raw answers, every DOI verdict and the graded miscitation file. The test cost $0.44. Ask and we will send the run files.
Is this the Claude Haiku 5.5 hallucination rate?
It is its citation hallucination rate: the share of DOIs it gave that no registry has issued, with no web access. A general accuracy score would need a different test. We measure citations because the answer is a fact anyone can check.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.