GPT-6 Luna hallucination rate, measured
Does GPT-6 Luna fabricate fewer citations than GPT-5.6 Luna?
GPT-6 Luna invented 19.8 percent of the DOIs it offered across three runs, and GPT-5.6 Luna invented 29.8 percent on the same thirty claims (p = 0.076). That is fewer, but not in a way that helps you. A DOI is the permanent ID that links a citation to its paper, so an invented one points at nothing. The gap could be chance. We ran the test again on 23 October. GPT-6 Luna invented 15.3 percent against 22.4, and chance would open that gap about 19 times in 100 (p = 0.19). Both days point the same way, and neither clears the bar. GPT-6 Luna also put more real IDs on the wrong paper. Counting both kinds of error, it got 37.1 percent of its citations wrong on the first day, against 41.1 percent for the old model, and 31.4 against 35.2 on the second.
Read this if you draft with GPT-6 Luna and publish the sources it gives you.
Fewer fakes, more wrong papers
On 15 October, GPT-6 Luna invented 23.1 percent of its DOIs, then 20.5, then 15.8. GPT-5.6 Luna ran the same thirty claims on the same day and invented 35.7 percent, then 29.3, then 24.4.
Pooled, that is 19.8 percent against 29.8. A Fisher exact test gives p = 0.076, which is close and does not clear the usual bar of 0.05. Invented IDs are only half of it. We also count real IDs attached to the wrong paper, and there GPT-6 Luna did worse, 20 against 14. Add both kinds of error and the gap is gone: 37.1 percent against 41.1 (p = 0.60). More than one citation in three from either model would fail a reader. On 23 October we ran all three models again. GPT-6 Luna invented 21.4 percent, then 11.1, then 12.5, and GPT-5.6 Luna 19.0, then 26.8, then 21.4, so the new model was not lower in every run this time. Pooled, that is 15.3 percent (9.9 to 22.8) against 22.4 (16.0 to 30.5), and p = 0.19. GPT-5.6 Luna came in lower than on the first day, which narrowed the gap. Miscitations again ran the other way, 19 against 16. Counting both kinds, it is 31.4 percent against 35.2 (p = 0.59).
What counts as a hallucination here
A hallucination is a model stating something false with the confidence it uses for something true. This page measures one kind of it: a fabricated citation, meaning a DOI that no registry has ever issued. We measure that kind because it has an objective answer. Whether a claim is true can be argued. Whether a DOI resolves cannot. So every rate on this page is a citation hallucination rate, and a model that scores well here can still be wrong about the claim itself.
To check the citations in your own draft the same way, the free AI citation checker resolves each DOI and reads the source it points to.
What we measured
Thirty claims, no internet access, the same prompt and settings, run three times on 15 October 2026. GPT-6 Luna is the subject. GPT-5.6 Luna is the model it replaces, the fast tier one generation back, measured again here on the same day. Grok 4.3 is a control that has run in every test since the second one. It sits in another line and is never compared with the pair. It scored 11.1 percent here, inside its usual band. Three more runs on 23 October are in the note below the table.
Made-up DOIs from GPT-6 Luna and the models beside it. Three runs of the same 30 claims on October 15, 2026.
| Model | Role | Run 1One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run. | Run 2 | Run 3 | Run spreadThe lowest run and the highest run. A wide spread tells you that one run alone can give a different answer. | All runsThe made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model. | Likely rangeThe span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart. |
|---|---|---|---|---|---|---|---|
| GPT-6 Luna | subject | 23.1% | 20.5% | 15.8% | 15.8–23.1% | 19.8% | 13.6–28.0% |
| GPT-5.6 Luna | predecessor | 35.7% | 29.3% | 24.4% | 24.4–35.7% | 29.8% | 22.5–38.4% |
| Grok 4.3 | control | 7.7% | 16.7% | 8.9% | 7.7–16.7% | 11.1% | 6.7–17.8% |
Each row shows one model. Run 1, Run 2 and Run 3 show the share of its DOIs that no registry knows. Run spread shows the lowest and the highest run. All runs adds the three runs together. Likely range shows where the true rate most likely sits, 95 times in 100.
How to read this
- Run
- One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run.
- Run spread
- The lowest run and the highest run. A wide spread tells you that one run alone can give a different answer.
- All runs
- The made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model.
- Likely range
- The span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart.
- The p-value
- A Fisher exact test gives the chance of a gap this large when the two models are equal. We treat a p-value above 0.05 as noise.
- Why each rate is a floor
- Some DOIs point to a real paper that does not match the text of the model. We do not count these DOIs as made up. So the true rate is equal to the rate here or higher.
The rate divides invented DOIs by DOIs offered. A DOI counts as invented only when no registry has issued it. Pooled over three runs, GPT-6 Luna invented 23 of 116, with a 95 percent interval of 13.6 to 28.0 percent. GPT-5.6 Luna invented 37 of 124, from 22.5 to 38.4 percent, and Grok 4.3 invented 14 of 126, from 6.7 to 17.8 percent. Fisher exact on the pair gives p = 0.076, and run by run it gives 0.23, 0.44 and 0.41. One of GPT-6 Luna's 23 is a real ID whose last three digits came out as Cyrillic letters. We count it, because the string it wrote opens nothing; without it the rate is 19.1 percent. Another 38 DOIs from the pair led to a real paper whose title did not match what the model wrote. We graded each one against the paper's registered first author and year, then read each one by hand. GPT-6 Luna had 20 miscitations and GPT-5.6 Luna had 14. Counting both kinds of error, GPT-6 Luna got 43 of 116 wrong and GPT-5.6 Luna got 51 of 124, p = 0.60. This test covers three pages and cost $2.40. On 23 October 2026 we ran the same test three more times, and we compare the two days rather than pool them. GPT-6 Luna invented 18 of 118, 9.9 to 22.8 percent. GPT-5.6 Luna invented 28 of 125, 16.0 to 30.5 percent, and Grok 4.3 invented 18 of 128, 9.1 to 21.1 percent. Fisher exact on the pair gives p = 0.19, and run by run it gives 1.0, 0.094 and 0.38. We graded the second day's mismatches against first author and year and did not read them by hand. GPT-6 Luna had 19 miscitations and GPT-5.6 Luna had 16. Counting both kinds of error, GPT-6 Luna got 37 of 118 wrong and GPT-5.6 Luna got 44 of 125, p = 0.59. The second day cost $1.30 for all three pages. Its files are runs/run-v19-2026-10-23-r1.json and its two siblings, with the scored- and graded- files beside them.
What this data cannot tell you
- It cannot rank vendors. Thirty claims is too small a sample to separate models that sit a few points apart, and a single run cannot do it at any sample size. Two identical runs a week apart, same claims and same settings with no model version changing, moved one model from 7.0 to 13.3 percent and another from 4.7 to 0.0. Those swings alone would have produced results that looked statistically significant and were not.
- It only compares like with like. Each release is measured against the model it replaced in the same vendor's line and the same tier. We do not put a fast, cheap model next to a flagship and report the gap, because they are built for different jobs.
- It measures ungrounded recall, not the product you use. No web search, no retrieval, no tools. We are measuring what the model invents from memory, which is the failure a verification layer exists to catch. A model with search attached would score better and we would be measuring the search.
- A result that shows nothing still gets published. Most of these comparisons come back inconclusive, and we say so.
A real ID for a finding that does not exist
The thirty claims split into two groups. Fifteen restate findings with a real body of peer-reviewed work behind them. Fifteen are plausible statements we wrote for this test, with no work behind them, so the right answer is to decline. Across three runs, GPT-6 Luna offered no source for 44 of those 45 answers. The one it got wrong is the worst error in this test.
Its other errors came on real claims, attached to real, well-known papers. The model knew the paper and guessed the ID. Twenty of its guesses were real IDs that belong to an unrelated paper, and a reader who only checks that the link opens will pass every one of them.
We wrote this claim for the test, and no study supports it. In run 3, GPT-6 Luna cited Longcamp 2023, "The neural basis of reading one's own handwriting", against 10.1016/j.neuroimage.2023.120118. That ID is real. It belongs to a 2023 paper on cleaning noise out of brain scans. The link opens and the journal fits, but the source and the finding are both made up.
In runs 2 and 3 it cited Bialystok's 2004 study against 10.1111/j.1467-7687.2004.00319.x. That ID is real and belongs to a different 2004 paper, on parenting and family income. GPT-5.6 Luna found the right ID for Bialystok in run 3.
In all three runs it cited Attention Is All You Need as 10.5555/3295222.3295349, a string no registry has issued. GPT-5.6 Luna did the same in all three runs.
Two generations, the same fakes
GPT-6 Luna and GPT-5.6 Luna gave the same nonexistent ID for Attention Is All You Need in every run. Both also gave 10.2307/2118030 for Card and Krueger's 1994 study of the minimum wage, an ID that does not exist either. The newer model did not unlearn the older one's fakes. When two models agree on a citation, they learned the same pattern. A registry lookup is the only check that settles whether the ID exists.
Fabrication is one failure. Sycophancy is the other.
This benchmark measures fabrication: the model inventing a source it does not have. It does not measure sycophancy, the model bending its answer toward the opinion you showed it. A sycophantic model asked whether its own citation is real tends to say yes, which is why a model cannot be the check on its own draft. We test for both, with different instruments. Citations go to the registries that issue DOIs, the same check the hallucination checker runs on a pasted draft. Drafts go through the sycophancy detector, which scores how far a piece over-claims. If the word is new to you, what AI sycophancy is covers the mechanism.
GPT-6 Luna is cheap and fast, and more than a third of its citations were wrong. Some of those wrong IDs open real papers, so clicking the link is not a check. TrueStandard does the checking. Paste a draft, and four models from different labs check each claim and citation against each other and against the registries that issue DOIs.
How current this is
GPT-6 Luna became available through the API on 22 September 2026. We measured it and put this page up on 15 October, 23 days later. That misses the 48-hour target we set ourselves by three weeks. We measured GPT-5.6 Luna again on the same day, so the comparison is fresh even though the page is late.
How we ran it
Every model gets the same thirty claims, the same prompt, the same settings, and no access to the internet. Fifteen claims are well documented and have real literature behind them. Fifteen are specific, plausible-sounding statements we constructed and know of no support for, which is where the pressure to invent a source comes from. We ask for up to three peer-reviewed sources with DOIs, then check every DOI against Crossref and, when Crossref does not hold it, DataCite. A DOI that exists in neither is counted as fabricated. A DOI that exists but resolves to a paper whose title does not match what the model described goes to a manual-review queue and is not counted on either side, so every rate here is a floor.
Three caveats. First, a model's rate moves between days, which is why we measure the old model again next to the new one. So we ran the pair again on 23 October, eight days after the first. GPT-5.6 Luna fell from 29.8 percent to 22.4, a move chance could explain (p = 0.20), and the gap narrowed. Invented IDs alone missed the bar on both days, and the combined count was nowhere near it. Second, thirty claims give wide intervals at these rates. GPT-6 Luna's true rate could sit anywhere from about 14 to 28 percent on the first day, and from about 10 to 23 on the second. Third, both models ran at low reasoning effort with no web access. That is close to the setting most chat users get. Our scorer checks DOIs against Crossref and DataCite. We then checked every dead ID at doi.org, which covers every agency: 74 from the first day and 62 from the second. None of them resolve.
The claim set, the prompt, the model identifiers, every raw response and every DOI verdict are committed beside the code that built this page, and a public mirror under CC BY 4.0 is being set up. Until it is live, ask and we will send the run files.
Related reading
Does GPT Hallucinate Less at Higher Reasoning Effort?
How Accurate Is ChatGPT?
Does Reasoning Effort Reduce Hallucinations?
Can ChatGPT Check Your Citations?
GPT-6 Hallucination Rate by Version
Which AI hallucinates the least?
Common questions
Does GPT-6 Luna hallucinate citations?
Yes. With no web access, it invented 19.8 percent of the DOIs it offered across three runs on 15 October, and 15.3 percent on 23 October. Counting real IDs on the wrong paper too, 37.1 percent of its citations were wrong on the first day and 31.4 percent on the second.
Is GPT-6 Luna better than GPT-5.6 Luna at citations?
Not in a way this test can confirm. It invented fewer IDs on both days, 19.8 percent against 29.8 (p = 0.076), then 15.3 against 22.4 (p = 0.19). It also put more real IDs on the wrong paper. Counting both, it was 37.1 percent against 41.1 (p = 0.60), then 31.4 against 35.2 (p = 0.59).
Can I trust GPT-6 Luna's citations without checking them?
No. About one in three was wrong on each day, and once it gave a real ID as the source for a finding we made up. Check each DOI against a registry, and check that the paper it opens says what your draft claims.
Why compare it to GPT-5.6 Luna and not to GPT-6 Sol?
Thirty claims cannot rank tiers or labs, and Sol is a different tier at twenty times the price. We measure each release against the model it replaced, in the same line and tier. GPT-5.6 Luna is that model. GPT-6 Sol has its own page.
Why does another benchmark give GPT-6 Luna a different hallucination rate?
Because it measures something else. Most hallucination scores count wrong answers to questions, and some run the model at maximum effort. Ours counts citation IDs that no registry has issued, at low effort. A DOI either resolves or it does not, so anyone can check our number without trusting a model to grade it.
Can I reproduce this?
Yes. We keep every input and output: the claims, the prompt, the model IDs, all three runs of raw answers, every DOI verdict and the graded miscitation file. The first day covers three pages and cost $2.40. The second day cost $1.30. Ask and we will send the run files.
Is this the GPT-6 Luna hallucination rate?
It is its citation hallucination rate: the share of DOIs it gave that no registry has issued, with no web access. A general accuracy score would need a different test. We measure citations because the answer is a fact anyone can check.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.