AI agent reliability is whether an agent gets the same task right every time, on tasks it has never seen. One passing run cannot show that. To test it, use tasks the builders never saw, run each one three times, and count a task as passed only when all three runs pass. A public test score is the best case. In a car assistant challenge Ayush Singh described on 15 October 2026, one agent passed 63.3 percent of the public tests and 50 percent of the hidden ones.
Repeat runs matter as much as hidden tasks. On 21 August 2026 we ran Gemini 3.7 Flash and Gemini 3.6 Flash on the same 30 claims three times in one afternoon, and the first run said the opposite of the next two. Below is what each test type tells you, what builders change to raise reliability, and five checks to run before you trust an agent's output.
A public test pass is the best case
Anyone can read a public test set, which makes it useful and also makes it weak. A team building an agent can look at the public tasks, see where the agent fails, and fix those cases. Each fix lifts the public score. None of it tells you how the agent does on a task nobody has seen.
So ask one question of any agent score: could the builders see these tasks? If they could, read the score as a ceiling.
What each agent test type tells you
| Test | What it shows | What it leaves open |
|---|---|---|
| One run, public tasks | The agent can do the task at least once | Whether the builders tuned it for these tasks |
| One run, hidden tasks | How it does on tasks the builders never saw | Whether it does the same thing next time |
| Three runs, all must pass | Whether it gets the task right every time | Tasks outside the test set |
| Your own tasks, three runs | How it does on your work, in your setup | Cases you did not think to write |
Read each row as one way to test an agent. The last column names the question that test leaves open.
The same test, run three times, gave three answers
Even a hidden test is one draw if you run it once. Language models do not give the same answer every time, and agents built on them carry that with them. We see it in our own model tests, which we run three times for every model.
On 21 August 2026 we gave Gemini 3.7 Flash and Gemini 3.6 Flash the same 30 claims, with the same settings, three times in one afternoon. A DOI, the permanent ID a paper carries, is how a citation points at that paper. A made-up one points at nothing. For each run we counted how many of the DOIs each model gave were made up.
Made-up DOIs per run, Gemini 3.7 Flash and Gemini 3.6 Flash, 21 August 2026
| Run | Gemini 3.7 Flash | Gemini 3.6 Flash | Lower rate |
|---|---|---|---|
| Run one | 11.1% | 4.4% | Gemini 3.6 Flash |
| Run two | 13.3% | 22.2% | Gemini 3.7 Flash |
| Run three | 8.9% | 15.6% | Gemini 3.7 Flash |
| All three runs | 11.1% | 14.1% | Too close to call |
Each row is one run of the same 30 claims on 21 August 2026. A lower number means fewer made-up DOIs. The last row adds all three runs together.
Run one gave Gemini 3.7 Flash the higher rate. Runs two and three gave it the lower one. Gemini 3.6 Flash scored 4.4 percent, then 22.2, then 15.6, so its worst run was five times its best. Nothing changed between the runs except the hour. Added up over all three runs on 21 August 2026, the two models sit at 11.1 and 14.1 percent, close enough that chance could explain the gap (p = 0.58). Source: our Gemini 3.7 Flash test page, which has the full run table.
Other models drift too. Claude Opus 5 ran the same 30 claims three times on 29 September 2026 and made up 2.2 percent of its DOIs, then 6.7, then 6.7, as our Claude Opus 5.5 page shows. In our 26 August 2026 test, re-running one fixed test moved a model's rate by between 1.3 and 7.2 points, as we showed in why hallucination benchmarks disagree. The run tables for each model are on our model release pages.
On 21 August 2026, one run of this test would have told you to keep Gemini 3.6 Flash. The next two runs said the reverse. An answer your agent gives you is a draw too, and you cannot see which way it fell. So TrueStandard runs four models on every claim in your draft at once and shows you where they disagree, in about 60 seconds.
Count a task as passed only when all three runs pass
Running every task three times is the answer to the one-draw problem. You run each task three times. The task counts as passed only if all three runs pass.
Compare that with a single run. An agent that gets a task right only some of the time can still pass it on one run. Under the three-run rule it fails that task, because it has to be right every time. The talk above draws the line against pass@3, which sounds the same and is much looser. Pass@3 counts a task if at least one of three runs succeeds. A task that passes once and fails twice clears pass@3 and fails pass^3.
Our runs on 21 August 2026 show why the rule matters. Across those three runs, Gemini 3.7 Flash made up a DOI on five of the 30 claims, and on only one of them in every run. Gemini 3.6 Flash slipped on nine claims and repeated two. Most misses showed up in one run and were gone in the next. One run shows you only that hour's misses. The three-run rule calls a claim safe only if it held up in all three. Source: our Gemini 3.7 Flash test page, which lists each claim.
Three runs is the least we would accept, and it is not proof. A task can pass three times and fail on the fourth. Run more when a mistake would cost you more.
Your setup is part of the score
An agent is more than its model. It has tools, prompts, memory and the rules that tie them together. Varad Vishwarupe, a PhD researcher at Oxford, puts it as a formula in an interview with Frank Stajano. How an agent behaves depends on three things: the model itself, what is built around it, and the people and setting it works with. He writes it as B = f(M, S, C). He argues that tests of the model alone cannot tell you how it will act once people use it.
So a score earned in one setup does not carry over to yours. Change the prompt, add a tool or give the agent a longer memory, and you have a new agent. Test the setup you will run, with the tasks you will give it.
Check the steps as well as the end result. In the car challenge above, a run could still fail even when the agent reached the right end state. Skipping a step or doing the steps in the wrong order was enough.
What builders change to make an agent more reliable
The agents that did best in the car challenge added structure around the model. In the same talk, published 15 October 2026, one entry checked each action the model proposed before letting it run. It retried a set number of times and then explained what it could not do. It passed 53.3 percent of the hidden set. Another had the model write a short program that keeps the steps in order across tool results, and it passed over 60 percent of the hidden set.
One rule from the talk applies to any agent: the record must hold what ran. A proposed action that a check turned down is not something the agent did. If you build agents, start there. Add a check before each action, cap the retries, and log what the tools did.
Five checks before you trust an agent
None of these needs a test lab. Each one uses tasks from your own work.
-
Run each task three times
Give the agent the same task three times, with the same settings. If the three results differ, no single score for that agent means much.
-
Use tasks it has never seen
Write a few tasks from your own work that the agent's builders could not have seen. Keep them private, so they stay hidden the next time you test.
-
Count a task only if all three runs pass
That is the three-run rule. A task that passes two runs out of three goes in the failed column, because it fails one time in three.
-
Check what it says against what it did
When the agent says it sent the email or changed the file, open the email or the file. The car challenge above scored agents on whether their claims of success matched what their tools did.
-
Give it a task it cannot do
Ask for something its tools cannot do, or leave out a detail it needs. A reliable agent says so. The car challenge had tasks built for this. In one, the tool that opens the sunroof shade was missing, and the agent had to explain that instead of claiming success.
If the agent writes text for you, its facts need the same treatment. Ask the question again in a second model and compare. Where the two disagree, check that claim by hand. Where they agree, you still have not proved it, because two models can repeat the same wrong answer. Our free compare AI answers tool puts GPT and Gemini side by side once a day. For what reliability means for plain factual answers, see AI reliability.
Notice what all five checks have in common: none of them takes one answer on trust. TrueStandard applies the same rule to the words your agent writes. Paste the draft, four models from different labs check every claim at once, and you see each claim they cannot agree on before your readers do.
Frequently Asked Questions
What is AI agent reliability?
It is how often an AI agent gets a task right when you run it again and again, on tasks it has not seen before. One passing run does not show it. Test on hidden tasks, run each task several times, and count a task as passed only when every run passes.
What is pass^k for AI agents?
Pass^k runs each task k times and counts it as passed only if all k runs pass. Pass^3 uses three runs. Pass@k sounds the same and is much looser, because it counts a task if any one of the k runs succeeds. An agent that gets a task right only some of the time passes pass@k and fails pass^k.
How many times should I run an AI agent test?
At least three, and count a task as passed only when all three pass. When we ran Gemini 3.7 Flash and Gemini 3.6 Flash on the same 30 claims three times on 21 August 2026, the first run said the opposite of the next two. Run more when a mistake would cost more.
Why do AI agents score lower on hidden test sets?
The builders cannot tune the agent for tasks they never see. Ayush Singh described a car assistant challenge in a talk on 15 October 2026. One agent in it passed 63.3 percent of the full public test set and 50 percent of the official hidden test set.
Is a public benchmark score useless for AI agents?
No. It shows the agent can do the task at least some of the time. Read it as the best case, and check whether the score came from one run or several.
Does a better model make an AI agent reliable?
Not on its own. How an agent behaves depends on the model, the tools and prompts around it, and the context it runs in. Test the whole agent in the setup you will use.
How do I check an AI agent's written output for false claims?
Check each claim against a second, independent model, and look at where they disagree. TrueStandard does this with four models from different labs in about 60 seconds, and lists every claim they cannot agree on.
Keep reading
Is There a Most Accurate AI Model?
The honest answer is no. The ranking changes with the task, the benchmark, and the month, and even the leader still hallucinates.
GPT-6 Astra Review: The First Week
Its 99.9% on ARC-AGI-3 came on OpenAI's own harness. On the neutral one it scored 62.7%.
Gemini 3.8 Flash Review: The Real Bill
It lists cheap and bills more: 1.77 times the output tokens of 3.7 Flash on the same suite.
How Accurate Is Jev? Jev 1.14 and 1.13 Tested on 108 Claims
On 108 claims checked against their sources, Jev 1.14 matched Jev 1.13's 96.3 percent in two of three runs, and scored 91.7 percent on the hardest ones, where two cheap chat models fell to the low eighties. Its calibration was no better than theirs.
TrueStandard vs Parafact
Both verify claims before you publish. The real difference is what one model can miss, and whether your long-form draft fits inside the check at all.
Check the claims in your next draft
An agent score is a number about someone else's test. The draft in front of you needs its own check. TrueStandard checks every claim in your draft across four models in about 60 seconds and shows you where they disagree.
TrueStandard is for writers who publish AI-assisted drafts. Paid plans check every claim and citation across four AI models before you hit publish.
Free: one check a day, no signup. Pro is $20 a month for 250 drafts.
Start Verifying →