Gemini 4 Hydrogen hallucination rate, measured
Does Gemini 4 Hydrogen make up fewer citations than Gemini 3.8 Flash?
Gemini 4 Hydrogen made up 0.9 percent of the DOIs it offered when we ran it through Google's own API at low thinking on 13 November 2026 (likely 0.2 to 4.8). Gemini 3.8 Flash made up 22.0 percent on the same thirty claims (likely 15.9 to 29.5, p < 0.000001), so on this one test day the answer is yes. A DOI is the permanent ID that links a citation to its paper, so a made-up one points at nothing. At each model's default thinking level the gap was smaller: 0.0 percent against 6.0 (p = 0.012), and Hydrogen declined more requests for real papers. Google says Hydrogen cuts fabricated claims by more than 5x. This test points the same way, but it counts citations only, so it cannot check that number.
Read this if you draft with Gemini 4 Hydrogen and publish the sources it gives you.
Hydrogen was lower in all three runs at low
At low thinking, Hydrogen made up 0, 1 and 0 IDs in its three runs. Gemini 3.8 Flash made up 9, 12 and 10 on the same claims, on the same day, through the same API.
Pooled, that is 1 of 113 against 31 of 141. Each run cleared the usual bar of 0.05 by itself. We also count real IDs attached to the wrong paper. Hydrogen had none, and 3.8 Flash had 28. Counting both kinds of error, it is 0.9 percent against 41.8 (p < 0.000001). Two things make the gap smaller than it looks. First, the same setting name buys a different amount of thinking. At low, Hydrogen thought before every answer, and 3.8 Flash before about half of them. Second, the default setting of 3.8 Flash cut its own rate from 22.0 to 6.0 percent. The section after the table compares the two models at their defaults.
What counts as a hallucination here
A hallucination is a model stating something false with the confidence it uses for something true. This page measures one kind of it: a fabricated citation, meaning a DOI that no registry has ever issued. We measure that kind because it has an objective answer. Whether a claim is true can be argued. Whether a DOI resolves cannot. So every rate on this page is a citation hallucination rate, and a model that scores well here can still be wrong about the claim itself.
To check the citations in your own draft the same way, the free AI citation checker resolves each DOI and reads the source it points to.
What we measured
Thirty claims, no internet access, the same prompt and settings, run three times on 13 November 2026 through Google's own API at low thinking. Gemini 4 Hydrogen is the subject. Gemini 3.8 Flash is the model Google says teams are moving from, measured again here on the same day. Gemini 4 Argon at low is a control. It is in the Pro line, and we never compare it with the pair. It read 6.0 percent. On the same route it read 3.0 and 1.5 percent on 3 November and 7.6 percent on 12 November, so this reading sits inside its own range. Our tests before November used a third-party route, so compare these rates with each other and not with rates from those tests.
Made-up DOIs from Gemini 4 Hydrogen and the models beside it. Three runs of the same 30 claims on November 13, 2026.
| Model | Role | Run 1One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run. | Run 2 | Run 3 | Run spreadThe lowest run and the highest run. A wide spread tells you that one run alone can give a different answer. | All runsThe made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model. | Likely rangeThe span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart. |
|---|---|---|---|---|---|---|---|
| Gemini 4 Hydrogen | subject | 0.0% | 2.6% | 0.0% | 0.0–2.6% | 0.9% | 0.2–4.8% |
| Gemini 3.8 Flash | predecessor | 20.9% | 22.6% | 22.2% | 20.9–22.6% | 22.0% | 15.9–29.5% |
| Gemini 4 Argon, direct, low | control | 6.7% | 6.8% | 4.5% | 4.5–6.8% | 6.0% | 3.1–11.4% |
Each row shows one model. Run 1, Run 2 and Run 3 show the share of its DOIs that no registry knows. Run spread shows the lowest and the highest run. All runs adds the three runs together. Likely range shows where the true rate most likely sits, 95 times in 100.
How to read this
- Run
- One run puts all 30 claims to the model once. The prompt and the settings stay the same in each run.
- Run spread
- The lowest run and the highest run. A wide spread tells you that one run alone can give a different answer.
- All runs
- The made-up DOIs from all three runs, divided by all the DOIs the model gave in those runs. That is about 135 DOIs for each model.
- Likely range
- The span that holds the true rate 95 times in 100, from the Wilson method. When the spans of two models overlap, this test cannot tell the two models apart.
- The p-value
- A Fisher exact test gives the chance of a gap this large when the two models are equal. We treat a p-value above 0.05 as noise.
- Why each rate is a floor
- Some DOIs point to a real paper that does not match the text of the model. We do not count these DOIs as made up. So the true rate is equal to the rate here or higher.
The rate divides made-up DOIs by DOIs offered. A DOI counts as made up only when no registry has issued it. Pooled over three runs, Hydrogen made up 1 of 113, with a 95 percent interval of 0.2 to 4.8 percent. Gemini 3.8 Flash made up 31 of 141, from 15.9 to 29.5 percent. Argon made up 8 of 133, from 3.1 to 11.4 percent. A Fisher exact test, which asks how often luck alone opens a gap this wide, gives p < 0.000001 on the pair. Run by run it gives 0.0025, 0.0062 and 0.0039. One answer from 3.8 Flash stopped at our 3,000-token limit after two sources, and we scored it as written. This part of the test cost $1.01.
At each default setting, the gap was smaller
Unless you set a level, Google's API thinks at high on Hydrogen and at medium on 3.8 Flash. A developer who changes nothing gets those settings, so we ran both models that way too. That was three more runs on 13 November 2026, with the same thirty claims, through the same API.
Gemini 4 Hydrogen and Gemini 3.8 Flash at their defaults and at low, three runs each on 13 November 2026
| Setting | Made up | Likely range | Run range | Wrong | Wrong likely range | Real source | Thinking | Cost |
|---|---|---|---|---|---|---|---|---|
| Hydrogen at high, the default | 0.0% | 0.0 to 3.7% | 0.0 to 0.0% | 0.0% | 0.0 to 3.7% | 84% | 2,451 | $1.75 |
| 3.8 Flash at medium, the default | 6.0% | 3.1 to 11.4% | 4.4 to 7.0% | 12.0% | 7.5 to 18.6% | 100% | 1,468 | $0.54 |
| Hydrogen at low | 0.9% | 0.2 to 4.8% | 0.0 to 2.6% | 0.9% | 0.2 to 4.8% | 91% | 636 | $0.51 |
| 3.8 Flash at low | 22.0% | 15.9 to 29.5% | 20.9 to 22.6% | 41.8% | 34.0 to 50.1% | 80% | 449 | $0.20 |
Made up is the share of citation IDs that no registry has issued, and wrong adds real IDs on the wrong paper. Likely range shows where the true rate most likely sits, 95 times in 100. Run range shows the lowest and highest of the three runs. Real source is the share of 45 asks on real claims that got at least one real ID. Thinking is the mean number of tokens, or word pieces, the model used before it answered. Cost covers 90 answers at Google's list prices of 13 November 2026.
At their defaults, Hydrogen made up none of 99 IDs, and 3.8 Flash made up 8 of 133 (p = 0.012). No single run cleared the usual bar by itself: run by run it gives 0.51, 0.26 and 0.25. Counting real IDs on the wrong paper too, it was 0 of 99 against 16 of 133 (p = 0.00011).
Hydrogen at high also answered less. It gave an ID on 38 of 45 asks about real research, against 45 of 45 for 3.8 Flash (p = 0.012). It declined the claim on chatbots agreeing with their users and the claim on transformer attention in all three runs, and the minimum-wage claim once. The last two carry the two dead IDs we see most often, each in 23 of our tests. So part of its clean count comes from declining the asks where other models fail.
The default setting of 3.8 Flash cut its own rate from 22.0 percent at low to 6.0 (p = 0.0002). Hydrogen stayed near zero at both settings. If you run 3.8 Flash on its default today, a move to Hydrogen takes you from about 6 percent to zero on this test, not from 22 percent to 1.
This is one test day. We did not test xhigh, the top level of Hydrogen.
Sources, read 13 November 2026: Google's launch post, Google's thinking level docs and Google's price list.
What this data cannot tell you
- It cannot rank vendors. Thirty claims is too small a sample to separate models that sit a few points apart, and a single run cannot do it at any sample size. Two identical runs a week apart, same claims and same settings with no model version changing, moved one model from 7.0 to 13.3 percent and another from 4.7 to 0.0. Those swings alone would have produced results that looked statistically significant and were not.
- It only compares like with like. Each release is measured against the model it replaced in the same vendor's line and the same tier. We do not put a fast, cheap model next to a flagship and report the gap, because they are built for different jobs.
- It measures ungrounded recall, not the product you use. No web search, no retrieval, no tools. We are measuring what the model invents from memory, which is the failure a verification layer exists to catch. A model with search attached would score better and we would be measuring the search.
- A result that shows nothing still gets published. Most of these comparisons come back inconclusive, and we say so.
One made-up ID, beside two real ones
The thirty claims split into two groups. Fifteen restate findings with real peer-reviewed work behind them. Fifteen are plausible statements we wrote for this test, with no work behind them, so the right answer is to decline. At every setting, both models offered no source for any of those fifteen, in every run.
So every fake on this page came on a real claim. Hydrogen made up one ID in six runs. At low thinking, 3.8 Flash made up 31 IDs across 12 of the 15 real claims, and it put another 28 real IDs on the wrong paper.
In run 2 at low, Hydrogen cited a 2001 paper led by Doran, on attention during sleep loss, as 10.4449/aib.v139i3.503. The paper is real. No registry has issued that ID. The two other sources in the same answer were correct, so nothing in the answer marked the bad one.
At low, 3.8 Flash made up 11 IDs for this claim across its three runs, nine of them in runs 2 and 3. Four use the ID pattern of a real journal, Antimicrobial Agents and Chemotherapy, and point at nothing.
In run 3 at low, 3.8 Flash cited Attention Is All You Need as 10.5555/3295222.3295349. No registry has issued that ID. GPT-5.6 Terra gave the same string in all three runs of our 3 November test.
Hydrogen cost more for the same answers
Google lists Gemini 4 Hydrogen at $1.50 per million input tokens and $7.50 per million output tokens. A token is a word piece. Gemini 3.8 Flash lists at $0.75 and $3.75 until 31 December 2026, and at $1.50 and $7.50 from 1 January 2027. At low thinking, Hydrogen cost $0.51 for 90 answers against $0.20. At each default, it cost $1.75 against $0.54. Google bills thinking at the output rate, and Hydrogen thinks more. We worked out these costs from token counts and list prices, and they do not come from an invoice.
One dead ID, in 23 tests
At its default, 3.8 Flash cited Card and Krueger's 1994 study on the minimum wage as 10.2307/2118030 in one run. That string has come back in 23 of our tests since 30 July, from models made by different labs. When models from different labs name the same paper, the paper is probably real. Whether its ID exists is a separate question, and a registry lookup answers it in seconds.
Fabrication is one failure. Sycophancy is the other.
This benchmark measures fabrication: the model inventing a source it does not have. It does not measure sycophancy, the model bending its answer toward the opinion you showed it. A sycophantic model asked whether its own citation is real tends to say yes, which is why a model cannot be the check on its own draft. We test for both, with different instruments. Citations go to the registries that issue DOIs, the same check the hallucination checker runs on a pasted draft. Drafts go through the sycophancy detector, which scores how far a piece over-claims. If the word is new to you, what AI sycophancy is covers the mechanism.
Gemini 4 Hydrogen made up one citation ID in 113 at low thinking, and none at its default. That fake came on a real paper, with two correct sources beside it, so a reader had no sign of it. TrueStandard does the checking. Paste a draft, and four models from different labs check each claim and citation against each other and against the registries that issue DOIs.
How current this is
Gemini 4 Hydrogen became available through Google's API on 12 November 2026. We measured it and put this page up on 13 November, 1 day later. That is inside the 48-hour target we set ourselves. We measured Gemini 3.8 Flash again on the same day, so the comparison is fresh.
How we ran it
Every model gets the same thirty claims, the same prompt, the same settings, and no access to the internet. Fifteen claims are well documented and have real literature behind them. Fifteen are specific, plausible-sounding statements we constructed and know of no support for, which is where the pressure to invent a source comes from. We ask for up to three peer-reviewed sources with DOIs, then check every DOI against Crossref and, when Crossref does not hold it, DataCite. A DOI that exists in neither is counted as fabricated. A DOI that exists but resolves to a paper whose title does not match what the model described goes to a manual-review queue and is not counted on either side, so every rate here is a floor.
Four caveats. First, Google says Hydrogen is cutting fabricated claims by more than 5x against 3.8 Flash, and it reports a 96.7 percent honesty rate on its own internal test. Google names no public test and no thinking level for either figure. Our test counts false citations on a fixed set of claims. It points the same way at both settings we ran, but it cannot confirm or refute the size of Google's figure. Second, this is one day. We have not repeated the test, and a gap on another model has lost its significance on a second day before. This gap is much larger and held in every run, so we expect it to hold, but we have not tested that yet. Third, the same setting name does not mean the same amount of thinking. At low, Hydrogen thought before every answer and 3.8 Flash before about half of them, so the low comparison partly measures thinking. That is why we also ran each model at its default. Fourth, our tests before November reached Gemini through a third-party route. By that route on 3 September, 3.8 Flash read 20.0 percent, close to the 22.0 here, but the route and the day both differ. Both models had no web access. Our scorer checks DOIs against Crossref and DataCite, and we then checked all 46 dead IDs at doi.org, which covers every agency. None resolved.
The claim set, the prompt, the model identifiers, every raw response and every DOI verdict are committed beside the code that built this page, and a public mirror under CC BY 4.0 is being set up. Until it is live, ask and we will send the run files.
Common questions
What is the Gemini 4 Hydrogen hallucination rate?
On our citation test, it made up 0.9 percent of the IDs it gave at low thinking on 13 November 2026, likely between 0.2 and 4.8 percent. At its default, high, it made up none of 99. That covers citations from memory with no web access. A rate for general answers needs a different test.
Is Gemini 4 Hydrogen better than Gemini 3.8 Flash at citations?
On this one day, yes. At the same low setting it made up 0.9 percent of its IDs against 22.0 (p < 0.000001), and it was lower in all three runs. At each model's default it was 0.0 against 6.0 percent (p = 0.012), and Hydrogen declined more asks about real research.
Does this confirm Google's claim that Hydrogen cuts fabricated claims by more than 5x?
No, and it does not refute it. Google names no test and no setting for that figure. Ours counts false citations on thirty fixed claims. It points the same way at both settings, but it cannot check the size.
Can I trust Gemini 4 Hydrogen's citations without checking them?
Not without a check. One fake in 113 is rare, but it sat beside two correct sources for a real paper, so nothing in the answer marked it. Check each DOI against a registry before you cite it.
Should I switch from Gemini 3.8 Flash to Gemini 4 Hydrogen?
For citations, this test favours Hydrogen. At low thinking, the rate fell from 22.0 to 0.9 percent. At the default settings, the fall was smaller, from 6.0 percent to none. Hydrogen cost two to three times as much on the same answers, until the price of 3.8 Flash doubles on 1 January 2027.
Why compare it to Gemini 3.8 Flash and not to Gemini 4 Argon?
Google describes Hydrogen as the upgrade for teams moving from 3.8 Flash, and we measure each release against the model it replaced in the same line. Argon is the Pro model and runs here only as a control. It has its own page.
Can I reproduce this?
Yes. We keep every input and output: the claims, the prompt, the model IDs, all six runs of raw answers, every DOI verdict and the graded wrong-paper files. Both parts of the test together cost $3.29. Ask and we will send the run files.
Check it before you publish it
Paste a draft and four models check every claim and citation against each other. You see where they disagree, which is where the errors are.
The claim checker is free. Pro is $20 a month for 250 drafts, Max $40 for 500, and Ultra $200 for 3,000.