Gemini Hallucination Rate by Version
By Arun Agrahri, Founder of TrueStandard. Published .
In our test of 3 November 2026, Gemini 4 Argon made up 3.0% of its citation IDs at low thinking and none of 110 at high. We ran it three times through Google's own API. Gemini 3.1 Pro made up 4.6% that day at low. No new Gemini has separated from the version it replaced on made-up IDs. The thinking setting moved the count more than the version did.
Key takeaways
- Gemini 4 Argon got 0 of 110 citation IDs wrong at high thinking in our test of 3 November 2026. At low it got 12 of 135 wrong, and at medium, the API default, 17 of 130.
- High thinking cost about ten times as much in that test: $1.46 for 90 answers against $0.14 at low. It gave an ID on 39 of 45 real claims, against 44 of 45.
- At low thinking in that test, Gemini 4 Argon made up 3.0% through Google's API and 6.3% through a third-party route (p = 0.24), too close to call.
- On 100 harder questions in our tests of 15 and 16 October 2026, Gemini 4 Argon made up 21.9%, then 17.6%. Gemini 3.8 Flash made up 42.5%, then 40.3%.
Each new Gemini against the version it replaced
We gave each pair the same thirty claims on the same day and asked for sources with DOIs. A DOI is the permanent ID a journal gives a paper, so a made-up one leads nowhere. Half the claims have real papers behind them, and half are ones we invented, where the right answer is to decline. The 3 November row ran through Google's own API at low thinking, and earlier rows ran through a third-party route.
Made-up citation IDs for each pair of Gemini versions, 2026
| Old to new | Tested | New | New likely range | Old | Old likely range | Luck | Result |
|---|---|---|---|---|---|---|---|
| 3.1 Pro to 4 Argon | 3 Nov 2026 | 3.0% | 1.2 to 7.4% | 4.6% | 2.1 to 9.6% | 0.54 | Too close to call |
| 3.1 Pro to 4 Argon | 15 Oct 2026 | 5.2% | 2.6 to 10.4% | 7.8% | 4.3 to 13.8% | 0.46 | Too close to call |
| 3.1 Pro to 4 Argon | 16 Oct 2026 | 5.3% | 2.6 to 10.5% | 6.2% | 3.2 to 11.7% | 0.80 | Too close to call |
| 3.7 Flash to 3.8 Flash | 3 Sep 2026 | 20.0% | 14.1 to 27.5% | 12.6% | 8.0 to 19.2% | 0.14 | Too close to call |
| 3.6 Flash to 3.7 Flash | 21 Aug 2026 | 11.1% | 6.8 to 17.5% | 14.1% | 9.2 to 20.9% | 0.58 | Too close to call |
New and Old show how many citation IDs no registry has issued, over three runs that day. Likely range is where the true rate most likely sits, 95 times in 100. Luck is the chance of a gap this wide if the two were equal, written as p in the text below. Under 0.05 is the usual bar. Compare inside one row only, because a rate can move between days.
Citation IDs Gemini 4 Argon got wrong at high thinking in our test of 3 November 2026, through Google's API. At medium, the API default, it got 17 of 130 wrong. Wrong means made up or pointing at the wrong paper.
Only high thinking cut Argon's wrong citations
In our test of 3 November 2026, we ran Argon at each thinking level it accepts. We counted two errors: made-up IDs and real IDs on the wrong paper. High got 0 of 110 wrong and low got 12 of 135 (p = 0.00067). Medium got 17 of 130 (p = 0.000019 against high). On made-up IDs alone, high against low is too close to call (p = 0.13). High also answered less. A real source came back on 87% of asks, against 96% at low (p = 0.27). This is one test day, so the result is strong and not yet settled.
Read every Gemini 4 Argon setting and run
Does Gemini hallucinate less with more thinking?
At medium, the API default, Gemini 4 Argon made up 3.1% of its IDs in our test of 3 November 2026, about one in thirty. TrueStandard checks each claim and citation across four models from different labs, and it flags what none of them can back up.
Try the free claim checkerGoogle's API and a third-party route gave similar rates
Our earlier tests reached Gemini through a third-party route. So in our test of 3 November 2026, we ran Argon both ways at low thinking. Google's API gave 3.0%, likely 1.2 to 7.4%, and the third-party route gave 6.3%, likely 3.3 to 12.0% (p = 0.24). Counting IDs on the wrong paper too, it was 8.9% against 12.7% (p = 0.42). This test cannot rule out a gap of a few points.
Flash versions stayed too close to call
In our test of 21 August 2026, Gemini 3.7 Flash made up 11.1% against Gemini 3.6 Flash's 14.1% (p = 0.58). In our test of 3 September, Gemini 3.8 Flash made up 20.0% against Gemini 3.7 Flash's 12.6% (p = 0.14). It scored higher in all three runs that day, and the gap still did not clear the bar.
Harder questions raise every Gemini rate
The thirty claims above lean on famous papers. We also asked 100 hard health and science questions from AA-Omniscience, a public question set from Artificial Analysis. Argon made up 21.9% on 15 October 2026 and 17.6% on 16 October. Those runs used low effort through a third-party route, so they are no test of high thinking.
Made-up citation IDs on 100 hard questions, October 2026
| Model | Tested | Answered | Made up | Likely range |
|---|---|---|---|---|
| Gemini 4 Argon | 15 Oct 2026 | 90% | 21.9% | 19.1 to 25.0% |
| Gemini 4 Argon | 16 Oct 2026 | 91% | 17.6% | 15.0 to 20.5% |
| Gemini 3.8 Flash | 15 Oct 2026 | 96% | 42.5% | 39.1 to 46.0% |
| Gemini 3.8 Flash | 16 Oct 2026 | 97% | 40.3% | 36.9 to 43.7% |
Answered is the share of questions with at least one citation ID. Made up is the share of those IDs that no registry has issued, over three runs that day. Likely range shows where the true rate most likely sits, 95 times in 100.
These rates cover memory alone, on a small test
We turned web search off, so these rates show what a model invents from memory. A model with search attached may score better, and then we would be measuring the search. Thirty claims give wide ranges, so small gaps can come from luck. We run each test three times because one run can point the wrong way. Most of these comparisons come back too close to call, and we publish those too.
Common questions
What is the Gemini 4 Argon hallucination rate?
It depends on the thinking level. In our test of 3 November 2026, Gemini 4 Argon made up 3.0% of its IDs at low, 3.1% at medium and 0.0% at high, through Google's API. In our tests of 15 and 16 October, at low through a third-party route, it made up 5.2% and 5.3%.
What is the Gemini hallucination rate?
It depends on the version and the question. In our 2026 tests on thirty claims, versions ran from 3.0% for Gemini 4 Argon on 3 November to 20.0% for Gemini 3.8 Flash on 3 September, at low effort. On 100 harder questions in October, they ran from 17.6% to 42.5%.
Is Gemini 3.8 Flash good?
For citations from memory, check its work. In our test of 3 September 2026, Gemini 3.8 Flash made up 20.0% of its IDs on thirty claims. In our October tests on 100 harder questions, it made up 42.5% and 40.3%. This test covers citations only, so it says nothing about other tasks.
Does Gemini hallucinate less with more thinking?
At the top level, on two models so far. In our September 2026 test, Gemini 3.7 Flash made up 13.3% at minimal and 4.5% at high. In our test of 3 November 2026, Gemini 4 Argon got 12 of 135 IDs wrong at low and 0 of 110 at high.
Who ran this test
Arun Agrahri, Founder of TrueStandard, runs this benchmark. No AI model scores it. Each citation ID is looked up in Crossref, DataCite and doi.org, and a person reads every odd result.
About TrueStandardCheck every citation before you publish
At its default setting in our test of 3 November 2026, Gemini 4 Argon made up about one ID in thirty. Paste a draft and four models from different labs check each claim and citation.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.