Every page written about TypeSafe's Jev repeats one claim: it returns a calibrated probability, and an ordinary language model does not. We measured that on 108 claims. It does not hold.
We ran Jev against two cheap chat models across six domains and four levels of difficulty. The calibration gap the coverage describes is not there. A different one is, and it shows up only on the claims that are hard.
The short answer
On easy claims all three models are perfect and price is the only thing separating them. On the hardest claims, the ones that look supported and are not, Jev holds 91.7 percent while both chat models fall into the low eighties. That is the real difference, and it is not calibration.
Expected calibration error came out at 0.066 for Jev, 0.061 for Gemini 3.1 Flash Lite and 0.067 for Claude Haiku 4.5. At 108 items those are one number. A chat model gives you a usable probability, once you ask for one properly.
108 claims checked against their source, six domains
| Model | Accuracy | Brier score | Calibration error | Cost |
|---|---|---|---|---|
| Jev 1.13 | 96.3% | 0.0331 | 0.0660 | $0.00033 |
| Gemini 3.1 Flash Lite | 94.4% | 0.0369 | 0.0611 | $0.00267 |
| Claude Haiku 4.5 | 93.5% | 0.0547 | 0.0669 | $0.00905 |
- Accuracy — how often it landed on the right answer.
- Brier score — how far its probabilities sat from what actually happened. Zero is perfect.
- Calibration error — when it says 80 percent, how close to 80 percent of those turn out to be true.
One run per model, 18 claims in each of six domains, 18 September 2026. Lower is better for the last two. Cost covers the whole 108-claim run.
What the coverage claims
Jev is not a chat model and it does not write text. You hand it source material and a set of questions you defined in advance, and it returns a typed answer and a probability for each one. The pitch is that those probabilities mean something: when it says 0.8, the thing is true about 80 percent of the time.
The coverage draws that against a language model asked for the same number. The standard line is that a language model says 0.95 about everything, because it produces a token shaped like a probability rather than reading one off a distribution. If that were true it would matter, because it decides whether you can put a threshold in your code and trust it.
TypeSafe does publish evaluation results, across four workflows, and its launch post of September 15 2026 is straightforward about the answer key: the reference is the average of two frontier models, Astra and Fable, rather than ground truth. Those evaluations therefore compare each model against two others, not against what is actually correct. What appears nowhere is a calibration figure: no reliability curve, no expected calibration error, no Brier score.
So the most repeated claim about this model had nothing published behind it. We have written before about why a confident answer from a model is not a correct one. This is that question with a number attached.
What the three numbers mean
Three numbers run through every table below. They measure different things, and a model can look strong on one while being useless on another.
Accuracy
Did it get the yes or no right?
Count the answers it got right, divide by the total. Accuracy is blind to how sure the model was: a call made at 51 percent confidence and one made at 99 percent both count as one correct answer. That makes it useful and incomplete: it cannot tell you when the model was guessing.
Brier score
How far were its probabilities from reality, on average?
For each answer, take the gap between the probability it gave and what happened, and square it. Say something is true and the model said 0.9: the gap is 0.1, squared is 0.01. Average that across every answer. Zero is perfect, lower is better. The squaring is the point, because it punishes a confident mistake far more than a hesitant one.
Calibration error
When it says 80 percent, is it right 80 percent of the time?
Written ECE, for expected calibration error, in the tables above and in most papers. Sort every answer into buckets by the confidence the model claimed, then check each bucket against what happened. Of everything it called 80 percent likely, was about 80 percent of it true? The average gap across the buckets is the calibration error. Lower is better.
Accuracy tells you whether to trust the answer. Calibration tells you whether to trust the number attached to it. A model can be accurate and badly calibrated, which means it is usually right but its confidence tells you nothing about when it is not. It can also be well calibrated and mediocre, which means it is wrong often and honest about it.
Calibration is the one you need if you plan to write a threshold into code. The whole appeal of a decision model is that you can say: act automatically above 0.9, send the rest to a person. That line is only safe if 0.9 means 0.9.
How the test was built
The task is grounding: here is a source passage, here is a claim, does the passage support the claim as stated? This is the shape of a citation check, and it is the failure that puts fabricated references into published work.
108 claims across six source passages: a quarterly financial report, a clinical trial summary, a product changelog, a travel policy, a supply contract and an engineering post-incident review. 56 supported, 52 not. We wrote and labelled every claim by hand before any model saw it.
We sorted the claims into four tiers, because an average over easy and hard items hides the thing worth knowing.
Four difficulty tiers
| Tier | What the claim does | Example against the finance passage |
|---|---|---|
| 1 · Literal | Restates the source almost word for word | "Revenue was $4.2M." |
| 2 · Paraphrase | Says the same thing in different words, or does simple arithmetic the source implies | "Churn increased by 0.7 percentage points." |
| 3 · Inference-adjacent | Follows from the source but is not written in it | "Revenue grew despite rising churn." |
| 4 · Adversarial near-miss | Looks supported and is not: invented causality, scope creep, percentage points read as percent, precision the source never gave | "The onboarding redesign caused the activation lift." |
Tier 4 is the tier that matters. A claim that a source does not actually support, but that reads as though it does, is the one that survives review and reaches a reader.
Each model got the whole task in one call per domain, and we asked for a probability rather than a verdict. We told the chat models to use the full range between zero and one. That instruction matters more than anything else in this study.
What 108 claims showed
Jev finished 1.9 points ahead of Gemini 3.1 Flash Lite and 2.8 ahead of Claude Haiku 4.5 on raw accuracy, with the best Brier score of the three. On ECE less than a hundredth separates them, which at 108 items is no separation at all.
Jev was never confidently wrong: not one answer on the wrong side of the line with extreme confidence attached. Both chat models did that once each. Jev also produced 34 distinct probability values across the run against Flash Lite's 10, so it discriminates more finely if you intend to threshold on the number.
The whole 108-claim run cost a third of a cent on Jev, eight times that on Flash Lite and 27 times that on Haiku. Jev was also about twice as fast.
Where this leaves the calibration claim
It does not survive. A cheap frontier-lab model, asked properly for a probability, calibrates about as well as the model built to produce calibrated probabilities. The claim repeated across every write-up of this launch had no measurement behind it, and the first measurement disagrees.
The gap is only on hard claims
Split the same 108 answers by difficulty and the averages come apart. On tiers 1 and 2 every model is perfect. Nothing distinguishes a purpose-built decision model from a cheap chat model when the claim is a restatement of the source.
Accuracy by difficulty tier
| Tier | Jev 1.13 | Flash Lite | Haiku 4.5 |
|---|---|---|---|
| 1 · Literal | 100% | 100% | 100% |
| 2 · Paraphrase | 100% | 100% | 100% |
| 3 · Inference-adjacent | 95.8% | 100% | 100% |
| 4 · Adversarial near-miss | 91.7% | 83.3% | 80.6% |
| Hardest tier, Brier score | 0.0595 | 0.0967 | 0.1358 |
On tier 4 the chat models lose eight and eleven points, and Jev loses four. Its Brier score on that tier is about half theirs, which says it hedged the tier-4 answers it got wrong instead of asserting them.
Jev is also the only model that dropped anything on tier 3, where it missed one claim the other two caught. It is not uniformly better. It is better where being wrong costs the most, and it gave up one claim in the tier where being wrong is cheap to catch.
That pattern is worth more than the headline average. A corpus that is mostly easy will tell you the three models are interchangeable. They are interchangeable on the easy part, and not on the part that reaches your readers.
The answer changed three times
This study started at 22 claims in one domain. It said Jev won on calibration by a wide margin, with Flash Lite emitting two distinct values across the whole run, 0.00 and 1.00. That looked like a clean result and it was wrong.
The cause was our own prompt. We had not asked Flash Lite to use the range. When we did, it started producing real probabilities and the gap closed. The finding had been measuring the quality of our instructions, not a property of the model.
Same question, three sample sizes
| Sample | What it covered | What it said |
|---|---|---|
| 22 claims | One domain, one prompt per model | Jev wins calibration decisively |
| 72 claims | Four domains, prompts corrected | Dead tie, 97.2 percent both |
| 108 claims | Six domains | Jev ahead 1.9 points overall, 8.4 on tier 4 |
Three sample sizes, three conclusions, one afternoon. The two domains we added last, a supply contract and an incident review, are where the chat models degraded and the tier-4 gap opened. Had we stopped earlier we would have published something we would now be retracting.
We describe our own near-miss because the alternative is pretending the first number was the final one. A benchmark that reverses twice under scaling is the normal case, and it is the reason a single run against a small corpus should not move anyone's decision.
Which one to use
If you are checking claims against sources at volume, Jev is the better instrument, for a narrower reason than the coverage suggests. It is not better calibrated. It is more accurate where accuracy is hard, it never attached high confidence to a wrong answer in this run, and it costs a fraction of the alternatives.
If your claims are mostly restatements of a source, any of the three will do and you should pick on price. That is a real finding and it is the less exciting half of this study.
None of the three can settle whether a claim is true in the world. This test asked only whether a passage supports a claim, a question whose answer sits in the source. Asked to adjudicate facts with no source, Jev declines, which is the right behaviour and a separate piece of work.
A probability from one model is one draw, and a threshold set from one run is a guess. We have written about what happens when one model is asked to check another, and we have watched four AI judges score dead sources as highly as live ones. The answer does not change because the checker got faster.
Frequently Asked Questions
How accurate is TypeSafe's Jev model?
On our test of 108 grounding claims across six domains, Jev answered 96.3 percent correctly, against 94.4 percent for Gemini 3.1 Flash Lite and 93.5 percent for Claude Haiku 4.5. The average hides the useful part: all three were perfect on literal and paraphrased claims, and the separation came entirely from adversarial near-misses, where Jev held 91.7 percent while the two chat models fell to 83.3 and 80.6.
Is Jev better calibrated than a language model?
Not in our measurement. Expected calibration error came out at 0.066 for Jev, 0.061 for Gemini 3.1 Flash Lite and 0.067 for Claude Haiku 4.5, which at 108 items is a three-way tie. The claim that language models cannot produce usable probabilities did not hold once we asked them explicitly to use the full range from zero to one.
What is expected calibration error?
ECE checks whether a model's stated confidence matches reality. Sort every answer into buckets by the confidence claimed, then compare each bucket to what actually happened: of everything the model called 80 percent likely, was about 80 percent of it true? The average gap across buckets is the ECE, and lower is better. It is a different question from accuracy, which only asks whether the answer was right.
What is the difference between Brier score and accuracy?
Accuracy counts right answers and ignores confidence, so a correct call made at 51 percent and one made at 99 percent score identically. Brier measures the squared gap between the probability given and what actually happened, averaged across all answers, so it penalises a confident mistake far more heavily than a hesitant one. A model can have good accuracy and a poor Brier score if it is right often but wildly overconfident when it is wrong.
Can Jev hallucinate?
Not in the sense of writing a false sentence, because it does not write sentences. It returns a value from a list you supplied, so it cannot invent an option that was not there. That is a guarantee about the shape of the output, not about whether the output is correct. Jev can return a valid answer that is wrong, and it did so four times on our hardest tier.
Can Jev replace an LLM for fact-checking?
For checking a claim against a supplied source, it performed better than both cheap chat models we tested, at a fraction of the cost. For deciding whether a claim is true without a source, no. Asked to adjudicate factual claims from its own knowledge, Jev flagged almost all of them as needing a live lookup rather than guessing, which is the right response and also a clear limit.
What accuracy does TypeSafe publish for Jev?
TypeSafe publishes workflow evaluations covering security incident response, agent-trace observability, invoice processing and customer service. Read the method before the number: its launch post states that the reference is the average of two frontier models, Astra and Fable, rather than ground truth, so those evaluations compare each model against two others rather than against what is actually correct. TypeSafe publishes no calibration figures at all, which is the gap this study fills.
How many claims does a grounding benchmark need?
More than most people run. Ours reversed its conclusion twice: at 22 claims it said Jev won calibration decisively, at 72 it said the field was tied, and at 108 across six domains it showed Jev ahead on accuracy and well ahead on hard claims. The two domains added last are where the gap appeared. A single small corpus will produce a confident answer and there is no way to tell from inside it whether that answer will survive.
How much does Jev cost compared to a small LLM?
Our entire 108-claim run cost $0.00033 on Jev, $0.00267 on Gemini 3.1 Flash Lite and $0.00905 on Claude Haiku 4.5. That is roughly eight times and 27 times more expensive respectively, for equal or lower accuracy on this task. Jev also finished in about half the wall-clock time.
How can I reproduce this test?
The six source passages and all 108 labelled claims were written by hand before any model saw them, and every model received the identical task on 18 September 2026. The important detail to copy is the instruction given to the chat models to use the full probability range, because omitting it is what produced our first and wrong result.
Keep reading
Is There a Most Accurate AI Model?
The honest answer is no. The ranking changes with the task, the benchmark, and the month, and even the leader still hallucinates.
How Accurate Is ChatGPT?
Accurate enough to trust for everyday questions, and wrong often enough to get you sued if you publish it unchecked. Here is what the measurements actually say, and what to do about it.
Why AI Is Confidently Wrong
Models sound certain every time, even when wrong. The confident tone you trust in people is worthless here. Here is the fix.
Should You Stop Using ChatGPT?
Researchers found AI made experts measurably worse on hard tasks. Here is when to trust ChatGPT, and when it is just telling you what you want to hear.
Does Claude Watermark My Writing?
Claude marks the words it chooses. If you wrote them, there is almost nothing to mark.
One model's probability is one draw
A threshold set from a single model's confidence is a guess about a distribution you have not seen. TrueStandard checks your draft against four or five models in parallel and shows you every claim they disagree on, which is where the errors actually live.
Start Verifying →