Gemini 4 Argon hallucination rate, measured
Does Gemini 4 Argon fabricate fewer citations than Gemini 3.1 Pro?
Gemini 4 Argon invented 5.2 percent of the DOIs it offered across three runs, and Gemini 3.1 Pro invented 7.8 percent on the same thirty claims (p = 0.46). That result is inconclusive. A DOI is the permanent ID that links a citation to its paper, so an invented one points at nothing. If the two models were equally good, chance would open a gap this wide about 46 times in 100. Argon did not measure worse. It still invented about one ID in nineteen. We ran the test again the next day, on 16 October, and got the same answer: 5.3 percent against 6.2, p = 0.80.
Read this if you draft with Gemini 4 Argon and publish the sources it gives you.
Within chance on both days
Argon invented 4.5 percent of its DOIs, then 4.4, then 6.7. Gemini 3.1 Pro ran the same thirty claims on the same day and invented 7.3 percent, then 9.1, then 7.0.
Pooled, that is 5.2 percent against 7.8. A Fisher exact test gives p = 0.46. That test asks how often chance alone would open a gap this size, and here the answer is often. In the third run the two models sat 0.3 points apart. So this is a null result: the new flagship did not cut the rate by enough for thirty claims to see. We also counted real IDs attached to the wrong paper. By that count, Argon got 14.9 percent wrong and Gemini 3.1 Pro got 18.8 percent (p = 0.41), which does not separate them either. On 16 October we ran the same test three more times. Argon invented 5.3 percent (2.6 to 10.5) and Gemini 3.1 Pro 6.2 percent (3.2 to 11.7), p = 0.80. This time Argon was higher in one run: 4 fakes in 44 IDs against 2 in 43. Counting both kinds of error, it got 13.6 percent wrong against 19.2 (p = 0.25). Argon's own rate barely moved, 5.2 percent then 5.3.
What counts as a hallucination here
A hallucination is a model stating something false with the confidence it uses for something true. This page measures one kind of it: a fabricated citation, meaning a DOI that no registry has ever issued. We measure that kind because it has an objective answer. Whether a claim is true can be argued. Whether a DOI resolves cannot. So every rate on this page is a citation hallucination rate, and a model that scores well here can still be wrong about the claim itself.
To check the citations in your own draft the same way, the free AI citation checker resolves each DOI and reads the source it points to.
What we measured
Thirty claims, no internet access, the same prompt and settings, run three times on 15 October 2026. Gemini 4 Argon is the subject. Gemini 3.1 Pro is the Pro model it replaces, measured again here on the same day. Grok 4.3 is a control that has run in every test since the second one. It sits in another line and is never compared with the pair. It scored 11.2 percent here, inside its usual band. Three more runs on 16 October are in the note below the table.
| Model | Run 1One complete pass of all thirty claims through the model. Same prompt, same settings, same day; only the hour changed. | Run 2 | Run 3 | RangeThe lowest and highest of the three runs. A wide range means one run on its own would have told you a different story. | Pooled (95% CI)All three runs added together, about 135 citations per model, with a 95 percent confidence interval: the span the true rate most likely sits in. When two models' spans overlap, this test cannot tell them apart. |
|---|---|---|---|---|---|
| Gemini 4 Argon subject | 4.5% | 4.4% | 6.7% | 4.4–6.7% | 5.2% (2.6–10.4) |
| Gemini 3.1 Pro predecessor | 7.3% | 9.1% | 7.0% | 7.0–9.1% | 7.8% (4.3–13.8) |
| Grok 4.3 control | 8.9% | 17.8% | 6.8% | 6.8–17.8% | 11.2% (6.9–17.6) |
How to read this
- Run
- One complete pass of all thirty claims through the model. Same prompt, same settings, same day; only the hour changed.
- Range
- The lowest and highest of the three runs. A wide range means one run on its own would have told you a different story.
- Pooled (95% CI)
- All three runs added together, about 135 citations per model, with a 95 percent confidence interval: the span the true rate most likely sits in. When two models' spans overlap, this test cannot tell them apart.
- The p-value
- A Fisher exact test asks how often chance alone would produce a gap this large at this sample size. Anything above 0.05 is treated as noise.
- Why each rate is a floor
- A DOI that points at a real paper the model described wrongly sits in a manual-review queue and is not counted either way, so the true rate can only be higher than the one shown.
The rate divides invented DOIs by DOIs offered. A DOI counts as invented only when no registry has issued it. Pooled over three runs, Argon invented 7 of 134, with a 95 percent interval of 2.6 to 10.4 percent. Gemini 3.1 Pro invented 10 of 128, from 4.3 to 13.8 percent, and Grok 4.3 invented 15 of 134, from 6.9 to 17.6 percent. Fisher exact on the pair gives p = 0.46, and run by run it gives 0.67, 0.43 and 1.0. Another 36 DOIs from the pair led to a real paper whose title did not match what the model wrote. We graded each one against the paper's registered first author and year, then read each one by hand. Argon had 13 miscitations and Gemini 3.1 Pro had 14. Counting both kinds of error, Argon got 20 of 134 wrong and Gemini 3.1 Pro got 24 of 128. The whole test cost $1.45. On 16 October 2026 we ran the test three more times, and we compare the two days rather than pool them. Argon invented 7 of 132, 2.6 to 10.5 percent. Gemini 3.1 Pro invented 8 of 130, 3.2 to 11.7 percent, and Grok 4.3 14 of 129, 6.6 to 17.4 percent. Fisher exact on the pair gives p = 0.80, and run by run it gives 0.68, 0.42 and 1.0. Counting both kinds of error, Argon got 18 of 132 wrong and Gemini 3.1 Pro 25 of 130, p = 0.25. The second day cost $1.41.
What this data cannot tell you
- It cannot rank vendors. Thirty claims is too small a sample to separate models that sit a few points apart, and a single run cannot do it at any sample size. Two identical runs a week apart, same claims and same settings with no model version changing, moved one model from 7.0 to 13.3 percent and another from 4.7 to 0.0. Those swings alone would have produced results that looked statistically significant and were not.
- It only compares like with like. Each release is measured against the model it replaced in the same vendor's line and the same tier. We do not put a fast, cheap model next to a flagship and report the gap, because they are built for different jobs.
- It measures ungrounded recall, not the product you use. No web search, no retrieval, no tools. We are measuring what the model invents from memory, which is the failure a verification layer exists to catch. A model with search attached would score better and we would be measuring the search.
- A result that shows nothing still gets published. Most of these comparisons come back inconclusive, and we say so.
Two fakes Gemini 3.1 Pro kept giving
The thirty claims split into two groups. Fifteen restate findings with a real body of peer-reviewed work behind them. Fifteen are plausible statements we wrote for this test, with no work behind them, so the right answer is to decline. All three models declined every one of those, in every run.
So every fake on this page came on a real claim, attached to a real, well-known paper. The model knew the paper but not its ID, and it wrote one anyway. Gemini 3.1 Pro's ten fakes are only five different IDs, because it kept repeating them.
Gemini 3.1 Pro cited Alhola's 2007 review of sleep loss as 10.2147/ndt.s286 in all three runs. No registry has issued that ID. The same model gave the same string for the same paper in our first test, on 30 July 2026. A fake that survives 77 days and every rerun is built into the model, so asking again will not catch it. On 16 October it gave the same string once more, in one run of three.
It cited Riebesell's 2000 paper in Nature as 10.1038/35036553, also in all three runs. The paper is real, and the ID points at nothing. On 16 October it made up an ID for this paper in every run again: the same string once, and twice a string one digit off, 10.1038/35036552, which also points at nothing.
Argon cited Card and Krueger's 1994 study as 10.1257/aer.84.4.772 in two of three runs. That ID does not exist, and models have now given it in thirteen of our tests. It was Argon's only repeat. Its seven fakes covered six claims. On 16 October its seven fakes were seven different IDs, and this one did not come back.
Two models, two made-up IDs for one real paper
Card and Krueger's 1994 paper on the minimum wage is real. In this test, Argon gave it one made-up ID and Grok 4.3 gave it another. Each string has now turned up in more than a dozen of our tests. Gemini 3.1 Pro did the same for Attention Is All You Need, with an ID we have now seen in fifteen tests. When two models name the same paper, the paper is probably real. Whether its ID exists is a separate question, and a registry lookup is the only way to answer it.
Fabrication is one failure. Sycophancy is the other.
This benchmark measures fabrication: the model inventing a source it does not have. It does not measure sycophancy, the model bending its answer toward the opinion you showed it. A sycophantic model asked whether its own citation is real tends to say yes, which is why a model cannot be the check on its own draft. We test for both, with different instruments. Citations go to the registries that issue DOIs, the same check the hallucination checker runs on a pasted draft. Drafts go through the sycophancy detector, which scores how far a piece over-claims. If the word is new to you, what AI sycophancy is covers the mechanism.
Argon makes up about one ID in nineteen, and on two days the model it replaces measured about the same. Some fakes come back on every run, and one came back after 77 days, so asking twice will not catch them. TrueStandard does the checking: paste a draft, and four models from different labs check each claim and citation against each other and against the registries that issue DOIs.
How current this is
Gemini 4 Argon became available on 13 October 2026. We finished measuring it on 15 October, 41 hours later. That is inside the 48-hour target we set ourselves.
How we ran it
Every model gets the same thirty claims, the same prompt, the same settings, and no access to the internet. Fifteen claims are well documented and have real literature behind them. Fifteen are specific, plausible-sounding statements we constructed and know of no support for, which is where the pressure to invent a source comes from. We ask for up to three peer-reviewed sources with DOIs, then check every DOI against Crossref and, when Crossref does not hold it, DataCite. A DOI that exists in neither is counted as fabricated. A DOI that exists but resolves to a paper whose title does not match what the model described goes to a manual-review queue and is not counted on either side, so every rate here is a floor.
Four caveats. First, earlier tests have shown a model's rate move between days. That is why we measure the old model again next to the new one, and why we ran this pair again a day later. Argon held steady, at 5.2 percent and then 5.3. Gemini 3.1 Pro went from 7.8 to 6.2 percent, a shift chance could explain (p = 0.63). Second, thirty claims give wide intervals. Argon's true rate could sit anywhere from about 3 to 10 percent, and that range overlaps most of Gemini 3.1 Pro's. Our grading is also lenient in Argon's favour. Five of its title mismatches had the right author, year and ID under a title that was not the paper's, and we did not count them as wrong. Counting them, Argon's error rate is 21.6 percent against 19.5 for Gemini 3.1 Pro, still too close to call. On the second day four of its matches were lenient in the same way. Counting those, it is 16.7 percent against 19.2 (p = 0.63). Third, both models ran at low reasoning effort with no web access. At that setting Argon reported no reasoning at all, so a higher setting is a different test. It was also cheap: Argon cost about an eighth of what Gemini 3.1 Pro cost on the same answers. Fourth, Gemini 3.1 Pro hit our 3,000-token answer limit on one claim, twice on each day, and stopped mid-list. We scored those answers as written, and dropping them leaves the result where it was. Our scorer checks DOIs against Crossref and DataCite. We then checked every dead ID at doi.org, which covers every agency: 23 from the first day and 26 from the second. None of them resolve.
The claim set, the prompt, the model identifiers, every raw response and every DOI verdict are committed beside the code that built this page, and a public mirror under CC BY 4.0 is being set up. Until it is live, ask and we will send the run files.
Common questions
Does Gemini 4 Argon hallucinate citations?
Yes. With no web access, it invented 5.2 percent of the DOIs it offered across three runs, about one in nineteen. Gemini 3.1 Pro invented 7.8 percent on the same claims that day. The gap is too small to call (p = 0.46). A day later it was 5.3 percent against 6.2 (p = 0.80).
Is Gemini 4 Argon better than Gemini 3.1 Pro at citations?
Not on this test, on two days. On 15 October it was 5.2 percent against 7.8 (p = 0.46). On 16 October it was 5.3 against 6.2 (p = 0.80), and Argon was higher in one of the three runs. Counting real IDs attached to the wrong paper too, it was 14.9 against 18.8 percent, then 13.6 against 19.2. A stricter grading puts Argon slightly behind on the first day and slightly ahead on the second.
Can I trust Gemini 4 Argon's citations without checking them?
No. About one ID in nineteen was invented, and about one in ten was real but pointed at the wrong paper. Check each DOI against a registry before you cite it.
Why compare it to Gemini 3.1 Pro and not to Gemini 3.8 Flash or GPT?
Thirty claims cannot rank labs, and Flash is a different tier. We measure each release against the model it replaced, in the same line and tier. Gemini 3.1 Pro is that model. Gemini 3.8 Flash has its own page.
Why does another benchmark give Gemini 4 Argon a different hallucination rate?
Because it measures something else. Most hallucination scores count wrong answers to questions. Ours counts citation IDs that no registry has issued. A DOI either resolves or it does not, so anyone can check our number without trusting a model to grade it.
Can I reproduce this?
Yes. We keep every input and output: the claims, the prompt, the model IDs, all three runs of raw answers, every DOI verdict and the graded miscitation file. The first day cost $1.45 and the second $1.41. Ask and we will send the run files.
Is this the Gemini 4 Argon hallucination rate?
It is its citation hallucination rate: 5.2 percent of the DOIs it gave in our three runs on 15 October 2026, and 5.3 percent on 16 October, had never been issued by any registry. A general accuracy score would need a different test. We measure citations because anyone can look up a DOI and check our number.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.