Does Grok 4.5 Fabricate Fewer Citations Than 4.3?

The result

We ran both Grok generations on the same 30 claims, same settings, same scoring pass. Each emitted exactly 45 DOIs, which makes the comparison very clean.

Grok 4.3 vs Grok 4.5, 30 claims, ungrounded, 2026-08-03

Fabrications fell from 5 to 2, and resolved citations rose from 28 to 43. The manual-review queue went from 12 to zero; that queue holds DOIs that exist, but whose titles do not match what the model claimed. On every column, 4.5 looks better than 4.3.

Fabricated means the DOI exists in neither Crossref nor DataCite. Corrected 2026-08-21: this post originally reported 24.4% and 13.3%, scored against Crossref alone. But arXiv registers with DataCite, so real preprints were being counted as fabrications. The scorer now checks both, and each run has been re-scored. Full method, raw responses and each verdict: the original four-model benchmark.

Why we are not calling this an improvement

Five versus two sounds decisive. It is not. A Fisher exact test on 5/45 against 2/45 returns p = 0.434, which is well outside any usual threshold. With 30 claims, a difference this size turns up often enough, even if the two models are identical.

We could have written the other headline. "Grok 4.5 halves fabrication rate" is true as arithmetic, and it would travel further, too. It would also be the exact failure this benchmark exists to catch: a confident number with nothing underneath it. We would deserve to be quoted back at ourselves the first time someone checked.

This is the structural problem with model-release coverage. A launch-week turnaround only fits small sample sizes, far too small to separate models that are really close. So the coverage reports noise as signal. The fix is not to stop measuring, but to publish the interval next to the number. Let it be boring when it is boring.

The claim we had to withdraw

This section first argued that the shape of the change was more convincing than the rate. The reason: 4.5's failures were a strict subset of 4.3's. On the corrected scoring that is no longer true, and the argument does not survive it.

Correctly scored, Grok 4.3 fabricated on three claims (S07, S12 and S15), and Grok 4.5 on two (S11 and S15). It fixed S07 and S12, and broke S11, which its predecessor got right. That is a trade, not a nested improvement, and just what a model that had merely gotten luckier would give. The first version of this post said 4.5 broke nothing. That was an artifact of our scorer, which counted a real arXiv preprint as invented. We were wrong.

What fixing a claim looked like

Take ocean acidification and shell formation in marine calcifiers. On that claim, 4.3 gave one real DOI and two invented ones, while 4.5 gave three that all resolve.

The same pattern held on vitamin D and respiratory infection. There, 4.3 gave two fabrications, while 4.5 gave three clean, resolving sources. Those two fixes are real, and they survive the re-scoring. What does not survive is the claim that nothing broke in the other direction. Then there is S11, where 4.5 invented an identifier its predecessor got right. Set against that, here is the fair summary: the newer model is probably better, but the evidence for it is weaker than we first said.

Two models, one invented DOI

The most interesting thing in the run was not about Grok at all. One claim says minimum-wage increases have small short-run disemployment effects, and it broke all four models in our first study. On that claim, GPT-5.5 and Grok 4.5 each gave the same fabricated identifier, on their own.

Look at the structure: it is the American Economic Review prefix, then volume 84, issue 4, page 772. That is just what a real identifier for that literature would look like. Two models from different vendors completed the same pattern and landed on the same invention.

This is a limit worth naming on our own approach. Cross-checking works because independent models fail independently. But here both were completing the same citation format from the same widely-documented literature. They can fail identically, so agreement between models is not corroboration in that case. What settles it is resolving the identifier. That is why our benchmark scores DOIs against a registry rather than asking a model to grade another model. The pattern turned out to be more stable than we expected. We re-ran this claim set for the GPT-5.6 release, and both GPT generations invented the same two identifiers as each other. One of them was a fake Gemini had already produced here.

How we ran it

Thirty claims from a frozen set: 15 with substantial peer-reviewed literature, 15 constructed for the benchmark with none. Each model was asked for up to three peer-reviewed sources, each needing a title, first author, year and DOI. No web search, no grounding, no retrieval. The target is what the model produces from memory, the failure a verification layer exists to catch.

Temperature 0, low reasoning effort, 3,000 max tokens, uniform across both models. Both Grok generations ran inside the same arm on the same day. So nothing but the model version differs between the two rows.

Each DOI was extracted and resolved against the Crossref API. When Crossref does not hold the record, the check falls through to DataCite. There is no language model anywhere in the scoring path. This benchmark is about models fabricating detail: grade it with a model and it inherits exactly the bias it claims to measure. The DataCite half of that check was missing until 2026-08-21, which is why the numbers in this post changed.

150 calls across five models, zero errors, $1.14 in API spend. The prompt set, the raw responses and each verdict are committed to the repository.

FAQ

Keep reading

What we measured on this model

Each release page carries our own fabrication data for one model version, measured against the model it replaced in the same vendor's line.