AI Reliability

Does GPT-5.6 Fabricate Fewer Citations Than GPT-5.5?

We asked both generations for peer-reviewed sources on the same 30 claims, on the same day. We resolved each DOI they produced against a real registry. GPT-5.6 came out lower. Then we noticed something in the data that matters more than which model won.

Does GPT-5.6 Fabricate Fewer Citations Than GPT-5.5?

Maybe, but our test cannot confirm it. GPT-5.6 Sol fabricated 3 of 39 DOIs against GPT-5.5's 5 of 45, a lower rate that is not statistically significant (p = 0.719). A DOI is the permanent ID that links a citation to its paper. We keep a frozen set of 30 claims and a script that looks up each DOI a model gives in Crossref, a DOI registry. So a new release becomes one command and a data point, and we skip the round of impressions.

There is a second reason we ran this one. GPT-5.6 Sol has been callable on OpenRouter since 9 July, and OpenRouter is the gateway our benchmark runs through. We rebuilt our benchmark on 3 August, and it still measured GPT-5.5 as OpenAI's current model, 25 days after its successor shipped. So we published a fabrication rate for a replaced model, again. The same thing happened with Grok 4.3 in our first study. A fix that comes 32 days late should not be hidden, so it opens this post instead of sitting in a footnote.

The result

Both GPT generations ran on the same 30 claims, with the same settings and the same scoring pass. Both ran inside a single arm on 10 August. Only the model version differs between these two rows.

GPT-5.5 and GPT-5.6 Sol, 30 claims, no web search, 10 August 2026

Model DOIs given Real Made up Made-up share
GPT-5.6 Sol 39 33 3 7.7%
GPT-5.5 45 39 5 11.1%

Each row is one model. DOIs given counts every DOI the model gave us. Real counts the DOIs that lead to a paper with the title the model gave. Made up counts the DOIs that neither Crossref nor DataCite knows. DataCite is the registry that issues DOIs for datasets and preprints. Made-up share divides made up by DOIs given.

GPT-5.6 offered six fewer citations and invented two fewer, so its fabrication rate is about a third lower. Stop reading here, and you would call the new model much safer. That is roughly what each launch-week write-up will tell you.

Made up means the DOI exists in neither Crossref nor DataCite. Full method, raw responses and each verdict: the original four-model benchmark.

Why this is not an improvement yet

Five against three sounds like progress. A Fisher exact test, a check for whether a gap between small counts is real, returns p = 0.719 on 5/45 against 3/39. A p-value shows how often chance alone opens a gap this size. This one is what you would expect if the two models behaved the same and we ran the test twice.

Here is the honest sentence. GPT-5.6 Sol fabricated fewer citations than GPT-5.5 in this run, and this test cannot show the difference is real.

When we published the Grok 4.5 result, we leaned on a second argument: 4.5's failures were a strict subset of 4.3's. We checked whether the same held here, and it did not. Re-scoring in August showed it had not held for Grok either, so we corrected that post too.

It does not.

Which claims each generation fabricated on

Model Fabricated on
GPT-5.5 S03, S11, S13, S15, U09
GPT-5.6 Sol S10, S11, S15

GPT-5.6 fixed three claims and broke one its predecessor got right. That is a trade, and a trade is the pattern you would see from a model that simply got luckier on the day. We are not reusing the Grok framing here, because the data does not support it.

This is the built-in problem with release-week coverage. The sample that fits in a fast turnaround is far too small to separate models that are really close. So the coverage reports noise as signal, and everyone repeats it. The fix is to keep measuring and publish the interval beside the number. Let the answer be boring when it is boring.

We watched the same pattern run again in September 2026, from the other side. OpenAI edited several benchmark figures in its own GPT-6 Astra launch post on the day it went up, and the archive caught it. What week one with Astra actually showed covers what survived checking and what did not.

What did get better

One change in the run is a difference in kind, and it is also the one that protects you.

Half our claim set is made up: specific, plausible statements we wrote for this benchmark, with no studies behind them. The correct behaviour is to decline. On all 15 of those, GPT-5.6 gave no DOI at all, and GPT-5.5 leaked on two.

GPT-5.5: invented a source for a claim we made up
U09 (fabricated 2023 neuroimaging finding) 10.1523/JNEUROSCI.2291-22.2023 — does not exist
GPT-5.6 Sol: same 15 claims
No DOI offered on any of them. 15 / 15 declined.

Two cells out of fifteen cannot carry a significance claim either. Still, "stops inventing sources when it has none" is a different behaviour from "invents slightly fewer." It is the one worth watching over the next few releases.

None of this changes what you have to do with the output. A 7.7% rate means roughly one in thirteen citations GPT-5.6 gives you still does not exist, and its interval runs to 20%. You cannot publish against that number without checking. It is the reason we built a tool that checks every claim against four models from different labs rather than trusting any one of them.

The finding that outranks the headline

We kept every model from our 3 August run. So this run doubles as something we had never done: a repeat of the exact same measurement. It used the same 30 claims, prompt, settings and scorer, seven days apart, and no model version changed in between.

If the metric is stable, every row should barely move.

Identical claims, identical settings, one week apart

Model 3 Aug 10 Aug
GPT-5.5 11.1% 11.1%
Grok 4.3 11.1% 13.3%
Grok 4.5 4.4% 8.9%
Claude Sonnet 5 7.0% 13.3%
Gemini 3.1 Pro 4.7% 0.0%

Each row is one model. Each cell gives the made-up share of its DOIs on the same 30 claims, in the run on that date in August 2026.

GPT-5.5 repeated exactly: 5 fabrications out of 45, both times. Claude nearly doubled, 3 to 6, and Gemini went from 2 to zero. Grok 4.3 moved 5 to 6, while Grok 4.5 moved 2 to 4. None of those models changed, and the questions did not change.

The obvious objection is that this says more about our benchmark than about the models. The swing is not limited to citation tests, or to us. A talk on building production voice agents came out the same week we ran this, and it gives a whole chapter to model drift. A text-to-speech model that scores well one week does worse the next. The talk puts that down to providers shipping updates on their own schedule.

That explanation does not cover our case. Both runs called pinned model versions through the same gateway, so there was no update to absorb. Either providers change something behind a version string that does not move, or a 30-claim sample is this noisy on its own. Under either reading, one run cannot stand as a measurement.

What that invalidates

This run produces a pairwise gap that clears the usual significance bar: Gemini against Claude at p = 0.027, and the same against Grok 4.3. Run the identical comparison on last week's data, and the gap does not exist. It depends entirely on two things. Claude sat at a one-week high and Gemini at a one-week low on the day we happened to measure.

So we are not publishing them as vendor findings, and neither should anyone quoting this page. A p-value from a single run knows nothing about the variance between runs. So it is more confident than the evidence deserves. We said earlier that 30 claims cannot rank vendors, and this run forced a stronger version on us. One run cannot rank vendors at any sample size.

This is also the cleanest argument we have for why picking the safest model is the wrong move. The gap between a model's good week and its bad week was larger than the gap between most of the models.

Two generations, one invented DOI

GPT-5.6 is not a copy of its predecessor. Still, it repeated two of its three fabrications exactly.

10.5555/3295222.3295349 · 10.2307/2118030
Both absent from Crossref and DataCite, verified 2026-08-10 and re-confirmed 2026-08-21, and emitted by GPT-5.5 and GPT-5.6 Sol alike.

A generation change did not disturb either one. They look like stable artifacts of how the family completes a citation pattern. The second ID is the exact string Gemini invented for the same claim in our previous run. That makes three model instances, two vendors and two generations, all giving one JSTOR record that does not exist.

One claim broke all four model families again: that minimum-wage increases cost few jobs in the short run. Each model invented something different, and each looked correct.

What four models invented for one claim

Model Made-up DOI
GPT-5.5 10.2307/2118030
GPT-5.6 Sol 10.2307/2118030
Grok 4.3 10.2307/2117956
Grok 4.5 10.1257/aer.84.4.772

Each row is one model on the minimum-wage claim. The Made-up DOI column shows the exact DOI the model gave. No registry knows any of them.

Look at the two JSTOR identifiers: 2118030 and 2117956 are neighbours in a real range. This is pattern completion over a citation format. That is why a fabricated DOI looks so convincing, and why "does it look right?" is not a check. It also names a limit on our own approach. Cross-checking works because independent models fail independently. When two of them complete the same pattern, they can fail identically. Looking up the ID is what settles it. That is why our benchmark scores DOIs against a registry instead of asking a model to grade another model.

How we ran it

We used thirty claims from a frozen set: 15 have solid peer-reviewed work behind them, and 15 we wrote for this benchmark have none. We asked each model for up to three peer-reviewed sources, each with a title, first author, year and DOI. There was no web search, grounding or retrieval. We test what a model produces from memory, because that is the failure a checking layer exists to catch.

All six used temperature 0 where the model accepts it, low reasoning effort, and 3,000 max tokens. A token is a piece of a word. GPT-5.5 and GPT-5.6 Sol ran inside the same arm on the same day, so nothing but the version number separates their rows.

We pulled out each DOI and looked it up in the Crossref API. No language model sits anywhere in the scoring path. This benchmark is about models inventing detail. Grade it with a model, and it inherits the exact bias it claims to measure.

We found one defect while writing this up. Our scorer's documentation claimed it checks a second registry, but the code only queried Crossref. We wrote at the time that we had manually resolved the eight fabricated DOIs quoted in this post. All returned 404, so the numbers stood. That reasoning was wrong, and we corrected it on 2026-08-21. The spot-check covered only the DOIs this post quoted as examples, and not the full set behind the rates. That full set held many arXiv IDs. Crossref does not index those, because arXiv registers with DataCite. So real preprints were being counted as fabrications. The scorer now checks both registries, and each run has been re-scored. GPT-5.5 and GPT-5.6's own rates in this post are unchanged, because neither cited arXiv on the claims they failed. Several other models in the comparison table above moved by as much as 11 points, and we updated the leaderboard. Checking the examples you chose to write about is not checking your data.

180 calls across six models, zero errors, $1.53 in API spend. The prompt set, the raw responses and each verdict are committed to the repository.

We ran the same kind of check against a public benchmark corpus later that month: what Claude Fable 5's top hallucination score does and does not tell you.

This test pinned reasoning effort at low. We later ran GPT-5.6 Luna at all five levels to see what effort changes: does GPT hallucinate less at higher reasoning effort.

FAQ

Does GPT-5.6 hallucinate less than GPT-5.5?

In our run, it fabricated 3 of the 39 DOIs it offered. GPT-5.5 fabricated 5 of 45, so that is 7.7% against 11.1%. Fisher's exact test returns p = 0.719, so this test cannot show the difference is real. The clearer gain is in behaviour: GPT-5.6 declined on all 15 claims we invented. GPT-5.5 fabricated a source for one of them.

Which AI model fabricates the fewest citations?

We do not think that question has a trustworthy answer yet, and our own data is why. We re-ran the identical 30 claims a week apart, and no model changed in between. Claude Sonnet 5 moved from 7.0% to 13.3%, and Gemini 3.1 Pro from 4.7% to 0.0%. The gap between one model's good week and its bad week was bigger than the gap between most of the models. So any single-run leaderboard is measuring the day as much as the model.

Why publish a result that is not statistically significant?

The alternative is publishing only the flattering half of what we measure, and a null result is still information. It tells you a launch-week comparison at this sample size cannot answer the question people are asking. The number, the interval, the p-value and the raw responses are all published. Anyone can re-run it at a larger n.

How many citations from GPT-5.6 should I expect to be fake?

Roughly one in thirteen of the ones it volunteers, based on this run. The 95% interval runs from 2.7% up to 20.3%, and it is wide because 30 claims is a small test. Either end of it is too high to publish against unchecked.

Will you re-run this on the next model release?

Yes, and faster than this one. The claim set is frozen and the run is a single command. So each frontier release should be a data point, not a debate. We were 32 days late here. It was the second time we published a rate for a model its vendor had already replaced. The next priority is repeated sampling across several days, since run-to-run swing is now clearly the bigger source of error.

Keep reading

What we measured on this model

Each release page carries our own fabrication data for one model version, measured against the model it replaced in the same vendor's line.

One in thirteen citations still does not exist. Check yours.

GPT-5.6 is the best model we have measured on this test. It still invented three sources out of thirty-nine. Two of them its predecessor invented before it. No model release removes the need to check, and picking a different vendor does not either. Paste your draft into TrueStandard, and four models from different labs check each claim in parallel. In about 60 seconds, they flag what none of them can confirm.

TrueStandard is for writers who publish AI-assisted drafts. Paid plans check every claim and citation across four AI models before you hit publish.

Free: one check a day, no signup. Pro is $20 a month for 250 drafts.

Check Your Draft →