Every page written about TypeSafe's Jev repeats one claim: it returns a calibrated probability, and an ordinary language model does not. We measured that on 108 claims. It does not hold.
We ran Jev against two cheap chat models across six domains and four levels of difficulty. The calibration gap the coverage describes is not there. A different one is, and it shows up only on the claims that are hard.
The short answer
On easy claims all three models are perfect and price is the only thing separating them. On the hardest claims, the ones that look supported and are not, Jev holds 91.7 percent. Both chat models fall into the low eighties. That is the real difference, and it is not calibration. Jev 1.14, released on 14 October, gets the same claims wrong as 1.13.
Expected calibration error came out at 0.066 for Jev, 0.061 for Gemini 3.1 Flash Lite and 0.067 for Claude Haiku 4.5. At 108 items those are one number. A chat model gives you a usable probability, once you ask for one properly.
108 claims checked against their source, six domains
| Model | Accuracy | Brier score | Calibration error | Cost |
|---|---|---|---|---|
| Jev 1.14 | 95.4–96.3% | 0.0265–0.0283 | 0.0781–0.0881 | $0.00033 |
| Jev 1.13 | 96.3% | 0.0331 | 0.0660 | $0.00033 |
| Gemini 3.1 Flash Lite | 94.4% | 0.0369 | 0.0611 | $0.00267 |
| Claude Haiku 4.5 | 93.5% | 0.0547 | 0.0669 | $0.00905 |
Jev 1.14: three runs on 16 October 2026, with the lowest and highest result across the three. Other rows: one run each on 18 September 2026. 18 claims in each of six domains. Lower is better for Brier score and calibration error. Cost covers the whole 108-claim run.
- Accuracy — how often it landed on the right answer.
- Brier score — how far its probabilities sat from what actually happened. Zero is perfect.
- Calibration error — when it says 80 percent, how close to 80 percent of those turn out to be true.
Is Jev 1.14 more accurate than 1.13?
Not on our test: we ran Jev 1.13 and 1.14 on the same 108 claims on 16 October 2026, three times each, and both got the same claims wrong. TypeSafe released 1.14 on 14 October and described it as small improvements to everything. Jev 1.14 scored 96.3 percent in two runs and 95.4 in the third, where one more answer landed exactly on the 50 percent line, which the cutoff counts as supported.
Jev 1.13 against Jev 1.14, same 108 claims, 16 October 2026
| Measure | Jev 1.13 | Jev 1.14 |
|---|---|---|
| Accuracy | 96.3% | 95.4–96.3% |
| Hardest tier, accuracy | 91.7% | 91.7% |
| Brier score | 0.0318–0.0322 | 0.0265–0.0283 |
| Hardest tier, Brier score | 0.0565–0.0579 | 0.0418–0.0451 |
| Calibration error | 0.0639–0.0647 | 0.0781–0.0881 |
| True claims scored 0.9 or higher | 49 of 56 | 44 of 56 |
| Cost per run | $0.00033 | $0.00033 |
Each row is one measure. Three runs per version. A range is the lowest and highest result across the three. Our 1.13 numbers from 18 September reproduced: the same accuracy, the same four wrong claims and the same cost.
Jev 1.14 is less sure of its answers, right and wrong. On its mistakes it now sits nearer the middle: it gave 0.57 to an unsupported claim about a memory alert, where 1.13 gave 0.76. Hedged mistakes cost less under the Brier score, so its Brier fell, most of all on the hardest claims. It also backs off on answers it gets right, from 0.94 to 0.88 on a correct revenue figure. A model that is right more often than it says it is gets a higher calibration error, and 1.14's rose.
We resampled the 108 claims to see which change holds. The Brier improvement is small enough to be chance, while the rise in calibration error is not, because 1.14 is underconfident on answers it gets right.
If your code acts on Jev's number
TypeSafe moves jev-latest to 1.14 a week after release, around 21 October 2026. Check your cutoff before you follow it. In our runs, an auto-accept line at 0.9 cleared 49 of the 56 true claims on 1.13 and 44 on 1.14. Those five go to your review queue instead. Pin the 1.13 model ID until you have rerun your own test set.
The multilingual gain
TypeSafe named one place to expect the largest gain: languages other than English. We translated the whole set, sources and claims, into Spanish, Japanese and Hindi. Every number, hedge and cause word was kept, and each translation was checked by translating it back. Then we ran both versions three times in each language.
Accuracy by language, 108 claims each
| Language | Jev 1.13 | Jev 1.14 | Hardest tier, 1.13 | Hardest tier, 1.14 |
|---|---|---|---|---|
| English | 96.3% | 95.4–96.3% | 91.7% | 91.7% |
| Spanish | 95.4% | 94.4–96.3% | 88.9% | 86.1–91.7% |
| Japanese | 92.6–93.5% | 91.7–92.6% | 83.3% | 80.6% |
| Hindi | 93.5% | 95.4% | 83.3% | 88.9% |
Each row is one language, three runs per version. A range is the lowest and highest result across the three runs. The hardest tier is tier 4, the near-miss claims.
The test set was machine-translated and checked by translating it back. One Japanese claim had turned 'more than two customers' into 'more than three', and we corrected it by hand. Text written natively in each language may behave differently.
The gain is mixed. Hindi improved by two claims, both on the hardest tier. Japanese got one claim worse. Spanish was level on accuracy, and its Brier score improved by a margin too large to be chance. Both versions are weaker outside English, and the drop is on the hardest claims. In our Japanese runs, Jev 1.14 got 80.6 percent of them right, against 91.7 in English.
What the coverage claims
Jev is not a chat model and it does not write text. You hand it source material and a set of questions you defined in advance. It returns a typed answer and a probability for each one. The pitch is that those probabilities mean something: when it says 0.8, the thing is true about 80 percent of the time.
The coverage draws that against a language model asked for the same number. The standard line is that a language model says 0.95 about everything. That is because it produces a token shaped like a probability rather than reading one off a distribution. If that were true it would matter, because it decides whether you can put a threshold in your code and trust it.
TypeSafe does publish evaluation results, across four workflows, and its launch post of September 15 2026 is straightforward about the answer key. The reference is the average of two frontier models, Astra and Fable, rather than ground truth. Those evaluations therefore compare each model against two others, not against what is actually correct. What appears nowhere is a calibration figure: no reliability curve, no expected calibration error, no Brier score.
So the most repeated claim about this model had nothing published behind it. We have written before about why a confident answer from a model is not a correct one. This is that question with a number attached.
What the three numbers mean
Three numbers run through every table below. They measure different things, and a model can look strong on one while being useless on another.
Accuracy
Did it get the yes or no right?
Count the answers it got right, divide by the total. Accuracy is blind to how sure the model was. A call made at 51 percent confidence and one made at 99 percent both count as one correct answer. That makes it useful and incomplete: it cannot tell you when the model was guessing.
Brier score
How far were its probabilities from reality, on average?
For each answer, take the gap between the probability it gave and what happened, and square it. Say something is true and the model said 0.9: the gap is 0.1, squared is 0.01. Average that across every answer. Zero is perfect, lower is better. The squaring is the point, because it punishes a confident mistake far more than a hesitant one.
Calibration error
When it says 80 percent, is it right 80 percent of the time?
Written ECE, for expected calibration error, in the tables above and in most papers. Sort every answer into buckets by the confidence the model claimed, then check each bucket against what happened. Of everything it called 80 percent likely, was about 80 percent of it true? The average gap across the buckets is the calibration error. Lower is better.
Accuracy tells you whether to trust the answer. Calibration tells you whether to trust the number attached to it. A model can be accurate and badly calibrated. That means it is usually right, but its confidence tells you nothing about when it is not. It can also be well calibrated and mediocre, which means it is wrong often and honest about it.
Calibration is the one you need if you plan to write a threshold into code. The whole appeal of a decision model is that you can say: act automatically above 0.9, send the rest to a person. That line is only safe if 0.9 means 0.9.
How the test was built
The task is grounding: here is a source passage, here is a claim, does the passage support the claim as stated? This is the shape of a citation check, and it is the failure that puts fabricated references into published work.
108 claims across six source passages. A quarterly financial report, a clinical trial summary, a product changelog, a travel policy, a supply contract and an engineering post-incident review. 56 supported, 52 not. We wrote and labelled every claim by hand before any model saw it.
We sorted the claims into four tiers, because an average over easy and hard items hides the thing worth knowing.
Four difficulty tiers
| Tier | What the claim does | Example against the finance passage |
|---|---|---|
| 1 · Literal | Restates the source almost word for word | "Revenue was $4.2M." |
| 2 · Paraphrase | Says the same thing in different words, or does simple arithmetic the source implies | "Churn increased by 0.7 percentage points." |
| 3 · Inference-adjacent | Follows from the source but is not written in it | "Revenue grew despite rising churn." |
| 4 · Adversarial near-miss | Looks supported and is not: invented causality, scope creep, percentage points read as percent, precision the source never gave | "The onboarding redesign caused the activation lift." |
Tier 4 is the tier that matters. A claim that a source does not actually support, but that reads as though it does, is the one that survives review and reaches a reader.
Each model got the whole task in one call per domain. We asked for a probability rather than a verdict. We told the chat models to use the full range between zero and one. That instruction matters more than anything else in this study.
What 108 claims showed
Jev finished 1.9 points ahead of Gemini 3.1 Flash Lite and 2.8 ahead of Claude Haiku 4.5 on raw accuracy. It also had the best Brier score of the three. On ECE less than a hundredth separates them, which at 108 items is no separation at all.
Jev was never confidently wrong: not one answer on the wrong side of the line with extreme confidence attached. Both chat models did that once each. Jev also produced 34 distinct probability values across the run against Flash Lite's 10. So it discriminates more finely if you intend to threshold on the number.
The whole 108-claim run cost a third of a cent on Jev. It cost eight times that on Flash Lite and 27 times that on Haiku. Jev was also about twice as fast.
Where this leaves the calibration claim
It does not survive. A cheap frontier-lab model, asked properly for a probability, calibrates about as well as the model built to produce calibrated probabilities. The claim repeated across every write-up of this launch had no measurement behind it, and the first measurement disagrees.
The gap is only on hard claims
Split the same 108 answers by difficulty and the averages come apart. On tiers 1 and 2 every model is perfect. Nothing distinguishes a purpose-built decision model from a cheap chat model when the claim is a restatement of the source.
Accuracy by difficulty tier
| Tier | Jev 1.13 | Flash Lite | Haiku 4.5 |
|---|---|---|---|
| Tier 1: Literal | 100% | 100% | 100% |
| Tier 2: Paraphrase | 100% | 100% | 100% |
| Tier 3: Inference-adjacent | 95.8% | 100% | 100% |
| Tier 4: Adversarial near-miss | 91.7% | 83.3% | 80.6% |
| Hardest tier, Brier score | 0.0595 | 0.0967 | 0.1358 |
Each cell gives the share of claims in that tier the model judged right. The last row gives the Brier score on tier 4, where lower is better. One run per model, 18 September 2026.
On tier 4 the chat models lose eight and eleven points, and Jev loses four. Its Brier score on that tier is about half theirs. That says it hedged the tier-4 answers it got wrong instead of asserting them.
Jev is also the only model that dropped anything on tier 3, where it missed one claim the other two caught. It is not uniformly better. It is better where being wrong costs the most. It gave up one claim in the tier where being wrong is cheap to catch.
That pattern is worth more than the headline average. A corpus that is mostly easy will tell you the three models are interchangeable. They are interchangeable on the easy part, and not on the part that reaches your readers.
The answer changed three times
This study started at 22 claims in one domain. It said Jev won on calibration by a wide margin, with Flash Lite emitting two distinct values across the whole run, 0.00 and 1.00. That looked like a clean result and it was wrong.
The cause was our own prompt. We had not asked Flash Lite to use the range. When we did, it started producing real probabilities and the gap closed. The finding had been measuring the quality of our instructions, not a property of the model.
Same question, three sample sizes
| Sample | What it covered | What it said |
|---|---|---|
| 22 claims | One domain, one prompt per model | Jev wins calibration decisively |
| 72 claims | Four domains, prompts corrected | Dead tie, 97.2 percent both |
| 108 claims | Six domains | Jev ahead 1.9 points overall, 8.4 on tier 4 |
Three sample sizes, three conclusions, one afternoon. The two domains we added last, a supply contract and an incident review, are where the chat models degraded and the tier-4 gap opened. Had we stopped earlier we would have published something we would now be retracting.
We describe our own near-miss because the alternative is pretending the first number was the final one. A benchmark that reverses twice under scaling is the normal case. It is the reason a single run against a small corpus should not move anyone's decision.
Which one to use
If you are checking claims against sources at volume, Jev is the better instrument. The reason is narrower than the coverage suggests. It is not better calibrated. It is more accurate where accuracy is hard. It never attached high confidence to a wrong answer in this run, and it costs a fraction of the alternatives.
If your claims are mostly restatements of a source, any of the three will do. You should pick on price. That is a real finding and it is the less exciting half of this study.
None of the three can settle whether a claim is true in the world. This test asked only whether a passage supports a claim, a question whose answer sits in the source. Asked to adjudicate facts with no source, Jev declines, which is the right behaviour and a separate piece of work.
A probability from one model is one draw, and a threshold set from one run is a guess. We have written about what happens when one model is asked to check another. We have also watched four AI judges score dead sources as highly as live ones. The answer does not change because the checker got faster. On paid plans, our AI fact checker puts four models from different labs on the same claim and shows you where they split.
Accuracy is one half of the case for a model like this. The other half is the speed claim, which the launch coverage repeats as 193x and nobody had timed. We did, and the answer ranges from 1.7x to 100x depending on what you put on the other side.
Jev 1.14 also reads images. We gave it 360 screenshots of AI search answers beside three vision models, and it answered 2,158 questions about them without a mistake.
The sentence above about declining is a measurement of its own. We gave it thirteen factual claims with the sources stripped off. It settled one and sent twelve for a live lookup.
Frequently Asked Questions
How accurate is TypeSafe's Jev model?
On our test of 108 grounding claims across six domains, Jev 1.13 answered 96.3 percent correctly, and Jev 1.14 matched it in two of three runs. That is against 94.4 percent for Gemini 3.1 Flash Lite and 93.5 percent for Claude Haiku 4.5. The average hides the useful part: all three were perfect on literal and paraphrased claims, and the separation came entirely from adversarial near-misses. Jev held 91.7 percent there, while the two chat models fell to 83.3 and 80.6.
Is Jev 1.14 more accurate than Jev 1.13?
Not on our test. We ran both versions three times each on 16 October 2026, on 108 grounding claims, and they got the same claims wrong. Jev 1.14 scored 96.3 percent in two runs and 95.4 in the third. It is less confident than 1.13: at a 0.9 threshold it cleared 44 of 56 true claims against 49. Its Brier score improved by a margin that could be chance, and its calibration error got worse by one that is not.
Is Jev 1.14 better in other languages?
Mixed. TypeSafe said the largest gain would be in languages other than English. On our test set translated into Spanish, Japanese and Hindi, Jev 1.14 got two more claims right in Hindi, one fewer in Japanese and the same number in Spanish. Both versions scored lower in all three languages than in English, mostly on the hardest claims.
Is Jev better calibrated than a language model?
Not in our measurement. Expected calibration error came out at 0.066 for Jev, 0.061 for Gemini 3.1 Flash Lite and 0.067 for Claude Haiku 4.5. At 108 items that is a three-way tie. The claim that language models cannot produce usable probabilities did not hold once we asked them explicitly to use the full range from zero to one.
What is expected calibration error?
ECE checks whether a model's stated confidence matches reality. Sort every answer into buckets by the confidence claimed, then compare each bucket to what actually happened. Of everything the model called 80 percent likely, was about 80 percent of it true? The average gap across buckets is the ECE, and lower is better. It is a different question from accuracy, which only asks whether the answer was right.
What is the difference between Brier score and accuracy?
Accuracy counts right answers and ignores confidence, so a correct call made at 51 percent and one made at 99 percent score identically. Brier measures the squared gap between the probability given and what actually happened, averaged across all answers. So it penalises a confident mistake far more heavily than a hesitant one. A model can have good accuracy and a poor Brier score if it is right often but wildly overconfident when it is wrong.
Can Jev hallucinate?
Not in the sense of writing a false sentence, because it does not write sentences. It returns a value from a list you supplied, so it cannot invent an option that was not there. That is a guarantee about the shape of the output, not about whether the output is correct. Jev can return a valid answer that is wrong, and it did so four times on our hardest tier.
Can Jev replace an LLM for fact-checking?
For checking a claim against a supplied source, it performed better than both cheap chat models we tested. It did that at a fraction of the cost. For deciding whether a claim is true without a source, no. Asked to adjudicate factual claims from its own knowledge, Jev flagged almost all of them as needing a live lookup rather than guessing. That is the right response, and also a clear limit.
What accuracy does TypeSafe publish for Jev?
TypeSafe publishes workflow evaluations covering security incident response, agent-trace observability, invoice processing and customer service. Read the method before the number. Its launch post states that the reference is the average of two frontier models, Astra and Fable, rather than ground truth. Those evaluations compare each model against two others rather than against what is actually correct. TypeSafe publishes no calibration figures at all, which is the gap this study fills.
How many claims does a grounding benchmark need?
More than most people run. Ours reversed its conclusion twice. At 22 claims it said Jev won calibration decisively, and at 72 it said the field was tied. At 108 across six domains it showed Jev ahead on accuracy and well ahead on hard claims. The two domains added last are where the gap appeared. A single small corpus will produce a confident answer. There is no way to tell from inside it whether that answer will survive.
How much does Jev cost compared to a small LLM?
Our entire 108-claim run cost $0.00033 on Jev, $0.00267 on Gemini 3.1 Flash Lite and $0.00905 on Claude Haiku 4.5. That is roughly eight times and 27 times more expensive respectively, for equal or lower accuracy on this task. Jev also finished in about half the wall-clock time.
How can I reproduce this test?
The six source passages and all 108 labelled claims were written by hand before any model saw them. Every model received the identical task on 18 September 2026, and both Jev versions were rerun on 16 October 2026. The important detail to copy is the instruction given to the chat models to use the full probability range. Omitting it is what produced our first and wrong result.
Keep reading
Is There a Most Accurate AI Model?
The honest answer is no. The ranking changes with the task, the benchmark, and the month, and even the leader still hallucinates.
Jev Images Tested Against Three Vision LLMs
We gave Jev 1.14 and three cheap vision models 360 screenshots of AI search answers. Jev and Claude Haiku 5.5 made no mistakes. Jev was 3 to 7 times faster and the cheapest per image.
How Accurate Is ChatGPT?
ChatGPT's accuracy rate depends on the question. It is reliable on well-documented basics, and independent benchmarks put frontier models, ChatGPT included, at 17 to 34 percent hallucination on factual claims.
Why AI Is Confidently Wrong
AI is confidently wrong because a model has no internal sense of when it is wrong. In one Carnegie Mellon task, chatbots showed about 90 percent confidence at about 65 percent accuracy.
TrueStandard vs FactCheckTool
These two tools look alike, but they solve opposite problems. One tells you if the media you read is fake. The other tells you if the draft you are about to publish is true.
One model's probability is one draw
A threshold set from a single model's confidence is a guess about a distribution you have not seen. TrueStandard checks your draft against four or five models in parallel and shows you every claim they disagree on. That is where the errors actually live.
TrueStandard is for writers who publish AI-assisted drafts. Paid plans check every claim and citation across four AI models before you hit publish.
Free: one check a day, no signup. Pro is $20 a month for 250 drafts.
Start Verifying →