GPT-6 Sol hallucination rate, measured
Does GPT-6 Sol fabricate fewer citations than GPT-5.6 Terra?
GPT-6 Sol invented 7.1 percent of the DOIs it offered across three runs, and GPT-5.6 Terra, the tier it replaces, invented 11.4 percent on the same thirty claims (p = 0.35). It fabricates fewer, but not by a margin we can confirm. A DOI is the permanent ID that links a citation to its paper, so an invented one points at nothing. Sol was lower in every run, but chance alone opens a gap this size about one time in three. Six of its seven fakes were the same two IDs, for two famous papers, and it gave both in every run.
Read this if you draft with GPT-6 Sol and publish the sources it gives you.
OpenAI has since replaced GPT-6 Sol with GPT-6.1 Sol. We measured the two side by side on the same claims, so the newer page also carries a fresh number for this model. Read the GPT-6.1 Sol measurement
Lower in every run, and still a null result
GPT-6 Sol invented 6.1 percent of its DOIs, then 9.1, then 6.1. GPT-5.6 Terra ran the same thirty claims on the same day and invented 10.0 percent, then 15.8, then 8.3.
Pooled, that is 7.1 percent against 11.4. A Fisher exact test gives p = 0.35, so chance alone could open a gap this size. We also count real IDs attached to the wrong paper. By that count, Terra got 19.3 percent of its citations wrong and Sol got 9.1 percent. That test gives p = 0.050, which sits on the line and does not clear it. Sol may cite better than Terra, and thirty claims on one day cannot show it.
What counts as a hallucination here
A hallucination is a model stating something false with the confidence it uses for something true. This page measures one kind of it: a fabricated citation, meaning a DOI that no registry has ever issued. We measure that kind because it has an objective answer. Whether a claim is true can be argued. Whether a DOI resolves cannot. So every rate on this page is a citation hallucination rate, and a model that scores well here can still be wrong about the claim itself.
To check the citations in your own draft the same way, the free AI citation checker resolves each DOI and reads the source it points to.
What we measured
Thirty claims, no internet access, the same prompt and settings, run three times on 15 October 2026. GPT-6 Sol is the subject. GPT-5.6 Terra held the same slot one generation back, the middle of OpenAI's three tiers, and was measured again here on the same day. Grok 4.3 is a control that has run in every test since the second one. It sits in another line and is never compared with the pair. It scored 11.1 percent here, inside its usual band.
| Model | Run 1One complete pass of all thirty claims through the model. Same prompt, same settings, same day; only the hour changed. | Run 2 | Run 3 | RangeThe lowest and highest of the three runs. A wide range means one run on its own would have told you a different story. | Pooled (95% CI)All three runs added together, about 135 citations per model, with a 95 percent confidence interval: the span the true rate most likely sits in. When two models' spans overlap, this test cannot tell them apart. |
|---|---|---|---|---|---|
| GPT-6 Sol subject | 6.1% | 9.1% | 6.1% | 6.1–9.1% | 7.1% (3.5–13.9) |
| GPT-5.6 Terra predecessor | 10.0% | 15.8% | 8.3% | 8.3–15.8% | 11.4% (6.8–18.5) |
| Grok 4.3 control | 7.7% | 16.7% | 8.9% | 7.7–16.7% | 11.1% (6.7–17.8) |
How to read this
- Run
- One complete pass of all thirty claims through the model. Same prompt, same settings, same day; only the hour changed.
- Range
- The lowest and highest of the three runs. A wide range means one run on its own would have told you a different story.
- Pooled (95% CI)
- All three runs added together, about 135 citations per model, with a 95 percent confidence interval: the span the true rate most likely sits in. When two models' spans overlap, this test cannot tell them apart.
- The p-value
- A Fisher exact test asks how often chance alone would produce a gap this large at this sample size. Anything above 0.05 is treated as noise.
- Why each rate is a floor
- A DOI that points at a real paper the model described wrongly sits in a manual-review queue and is not counted either way, so the true rate can only be higher than the one shown.
The rate divides invented DOIs by DOIs offered. A DOI counts as invented only when no registry has issued it. Pooled over three runs, GPT-6 Sol invented 7 of 99, with a 95 percent interval of 3.5 to 13.9 percent. GPT-5.6 Terra invented 13 of 114, from 6.8 to 18.5 percent, and Grok 4.3 invented 14 of 126, from 6.7 to 17.8 percent. Fisher exact on the pair gives p = 0.35, and run by run it gives 0.68, 0.49 and 1.0. Another 13 DOIs from the pair led to a real paper whose title did not match what the model wrote. We graded each one against the paper's registered first author and year, then read each one by hand. GPT-6 Sol had 2 miscitations and Terra had 9. Counting both kinds of error, Sol got 9 of 99 wrong and Terra got 22 of 114, p = 0.050. This test covers three pages and cost $2.40.
What this data cannot tell you
- It cannot rank vendors. Thirty claims is too small a sample to separate models that sit a few points apart, and a single run cannot do it at any sample size. Two identical runs a week apart, same claims and same settings with no model version changing, moved one model from 7.0 to 13.3 percent and another from 4.7 to 0.0. Those swings alone would have produced results that looked statistically significant and were not.
- It only compares like with like. Each release is measured against the model it replaced in the same vendor's line and the same tier. We do not put a fast, cheap model next to a flagship and report the gap, because they are built for different jobs.
- It measures ungrounded recall, not the product you use. No web search, no retrieval, no tools. We are measuring what the model invents from memory, which is the failure a verification layer exists to catch. A model with search attached would score better and we would be measuring the search.
- A result that shows nothing still gets published. Most of these comparisons come back inconclusive, and we say so.
Six fakes, two papers
The thirty claims split into two groups. Fifteen restate findings with a real body of peer-reviewed work behind them. Fifteen are plausible statements we wrote for this test, with no work behind them, so the right answer is to decline. GPT-6 Sol offered no source for any of the fifteen in any run, and gave sources for every real claim.
All seven of its fakes came on real claims. Six were the same two IDs, for two of the most cited papers in their fields, and it gave both in all three runs. A fake that comes back on every run is built into the model, so asking again will not catch it.
In all three runs, GPT-6 Sol cited Attention Is All You Need as 10.5555/3295222.3295349. That paper appeared at a conference whose proceedings carry no DOI, and no registry has issued this one. Every OpenAI model in this test gave the same string at least once.
In all three runs it cited Card and Krueger's 1994 study as 10.2307/2118030. It looks like a JSTOR record and does not exist. The same string has turned up in our tests since the first one, in July. GPT-6.1 Sol, the update, cited a real paper by the same authors instead.
Its one other fake. GPT-6 Sol cited Varady's 2013 trial of alternate-day fasting against 10.1186/1743-7075-10-97. The paper is real, and its ID is 10.1186/1475-2891-12-146. The model had the paper right and the ID wrong.
Six models, one fake ID
We ran six OpenAI models on 15 October, across two generations and three tiers. Every one of them gave 10.5555/3295222.3295349 for Attention Is All You Need at least once. When models agree on a citation, they learned the same pattern. That says nothing about whether the ID exists, and a registry lookup is the only check that settles it.
Fabrication is one failure. Sycophancy is the other.
This benchmark measures fabrication: the model inventing a source it does not have. It does not measure sycophancy, the model bending its answer toward the opinion you showed it. A sycophantic model asked whether its own citation is real tends to say yes, which is why a model cannot be the check on its own draft. We test for both, with different instruments. Citations go to the registries that issue DOIs, the same check the hallucination checker runs on a pasted draft. Drafts go through the sycophancy detector, which scores how far a piece over-claims. If the word is new to you, what AI sycophancy is covers the mechanism.
GPT-6 Sol makes few fakes, and the ones it makes come back on every run. Asking it twice will not catch them, and other OpenAI models give the same ones. TrueStandard does the checking. Paste a draft, and four models from different labs check each claim and citation against each other and against the registries that issue DOIs.
How current this is
GPT-6 Sol became available through the API on 22 September 2026. We measured it and put this page up on 15 October, 23 days later. That misses the 48-hour target we set ourselves by three weeks. We measured GPT-5.6 Terra again on the same day, so the comparison is fresh even though the page is late.
How we ran it
Every model gets the same thirty claims, the same prompt, the same settings, and no access to the internet. Fifteen claims are well documented and have real literature behind them. Fifteen are specific, plausible-sounding statements we constructed and know of no support for, which is where the pressure to invent a source comes from. We ask for up to three peer-reviewed sources with DOIs, then check every DOI against Crossref and, when Crossref does not hold it, DataCite. A DOI that exists in neither is counted as fabricated. A DOI that exists but resolves to a paper whose title does not match what the model described goes to a manual-review queue and is not counted on either side, so every rate here is a floor.
Three caveats. First, all three runs happened on one day. Earlier tests have shown a model's rate move between days, which is why we measure the old model again next to the new one. A second day for this pair is the open work on this page. Second, the pairing is a choice, and other benchmarks make the other one. OpenAI kept the name Sol and moved it down a tier. GPT-5.6 Sol was the flagship, and GPT-6 Sol sits below GPT-6 Astra at half the old price. We compare like with like, so the predecessor here is Terra, the old middle tier. We ran GPT-5.6 Sol on the same day too. It invented 7.2 percent of its DOIs, against 7.1 for GPT-6 Sol (p = 1.0). Third, all the models ran at low reasoning effort with no web access. Our scorer checks DOIs against Crossref and DataCite. We then checked all 74 dead IDs from this test at doi.org, which covers every agency. None of them resolve.
The claim set, the prompt, the model identifiers, every raw response and every DOI verdict are committed beside the code that built this page, and a public mirror under CC BY 4.0 is being set up. Until it is live, ask and we will send the run files.
Common questions
Does GPT-6 Sol hallucinate citations?
Yes. With no web access, it invented 7 of the 99 DOIs it offered across three runs, or 7.1 percent. Six of the seven were the same two IDs, which it gave in every run.
Is GPT-6 Sol better than GPT-5.6 Terra at citations?
It was lower in every run, and we cannot confirm the gap. It invented 7.1 percent of its DOIs, against 11.4 for Terra (p = 0.35). Counting real IDs on the wrong paper too, it got 9.1 percent wrong, against 19.3 (p = 0.050). Both results come from three runs on one day.
Why compare GPT-6 Sol with GPT-5.6 Terra and not GPT-5.6 Sol?
Because OpenAI moved the name down a tier. GPT-5.6 Sol was the flagship at $4 and $20 per million tokens. GPT-6 Sol sits below GPT-6 Astra at $2 and $10, the slot GPT-5.6 Terra held. We ran GPT-5.6 Sol on the same day anyway. It invented 7.2 percent of its DOIs, against 7.1 for GPT-6 Sol (p = 1.0).
Can I trust GPT-6 Sol's citations without checking them?
No. About one ID in fourteen was invented, and it gave the same fakes on every run, so asking again does not help. Check each DOI against a registry before you cite it.
Why does another benchmark give GPT-6 Sol a different hallucination rate?
Because it measures something else. Most hallucination scores count wrong answers to questions, and some run the model at maximum effort. Ours counts citation IDs that no registry has issued, at low effort. A DOI either resolves or it does not, so anyone can check our number without trusting a model to grade it.
Can I reproduce this?
Yes. We keep every input and output: the claims, the prompt, the model IDs, all three runs of raw answers, every DOI verdict and the graded miscitation file. The test covers three pages and cost $2.40. Ask and we will send the run files.
Is this the GPT-6 Sol hallucination rate?
It is its citation hallucination rate: the share of DOIs it gave that no registry has issued, with no web access. A general accuracy score would need a different test. We measure citations because the answer is a fact anyone can check.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.