Jev by TypeSafe: A New Agent Layer, If Calibration Holds

TypeSafe AI says its new model, Jev, is 193.6x faster and 444.6x cheaper than frontier LLMs. The most careful independent test we have found measured about 2.9x faster and about 12x cheaper. Both numbers are real. The distance between them, close to two orders of magnitude on speed, tells you most of what you need to know about this launch. The exception is the one thing nobody has measured yet: whether Jev’s probabilities mean what they say.

That gap is our argument. Jev is not an LLM competitor. It is a bid for a new layer in the agent stack: the fast, cheap, typed decision node. That covers the router, the triage step, the guardrail, the verifier and the “smart if-statement”, all of which agents currently implement by prompting a frontier model and parsing its prose. The architectural insight is real and overdue. Most agent “decisions” are classification problems wearing a generation costume, and we have been paying generation prices, generation latency and generation failure modes (parse errors, schema drift, retries) for them. But the category claim hangs on one unproven promise: calibration. “No type errors” is table stakes. A probability you can safely put a threshold on is the actual product. As of today, TypeSafe has published no reliability diagram, no expected calibration error (ECE) and no ablation.

First, the facts. TypeSafe AI, founded in 2024 in San Francisco, launched Jev in limited early access on September 15, 2026, alongside a $40M seed round led by DCVC, according to TechCrunch (September 18). CEO Diogo Almeida spent about four years at OpenAI working on RLHF, InstructGPT, ChatGPT and GPT-4. His co-founders are Erik Gafni and Sasha Sheng. Almeida’s launch tweet opens with “After co-inventing ChatGPT”, and TechCrunch’s headline calls him “a ChatGPT inventor.” That is their framing, not ours. RLHF predates his OpenAI work (Christiano et al., 2017), and ChatGPT was a large team effort. The launch landed hard regardless. Latent Space counted 36M views of the launch video in two days and estimated that TypeSafe had reached roughly 13% of teams.

View original post on X →

What makes Almeida’s background interesting is not a job title. He worked on the exact post-training stage that, by OpenAI’s own published measurement, damaged GPT-4’s calibration. We return to that below, because it is the strongest version of TypeSafe’s case and no other coverage we have seen draws the connection.

What Jev Actually Is, Technically

Start with what an agent does today when it needs a decision. Suppose a support-ticket router must pick one of five departments. You prompt an LLM, and it generates tokens one at a time, each token costing a full sequential forward pass, until it has written something like {"department": "billing"}. Your code then parses that string, validates it against a schema, and retries whenever the model wraps the JSON in prose or invents a sixth department. At the end you have a value. You do not have a trustworthy number for how likely that value is to be right.

Jev replaces that loop with a different contract. According to TypeSafe’s launch post, the input is unstructured text, structured program state and a predefined set of allowed outputs. The output is a typed value from that set, with calibrated probabilities and confidence scores for every allowed output, all produced in a single non-autoregressive query. MarkTechPost’s technical explainer (September 19) documents the API. A POST /v1/systemone call carries the state plus a typed questions map. There are three primitives: Choice (up to 255 options), Score (ordered levels) and Noul (a boolean probability). “Questions run in parallel and in isolation against the same state.” OpenRouter’s launch post (September 18) summed up the developer pitch: “There is no JSON prompting, parsing layer, and nothing to validate against.”

View original post on X →

TypeSafe calls this class of model “System One models”, borrowing Kahneman’s split between fast, intuitive System 1 thinking and slow, deliberate System 2 thinking. The company states the trade-off plainly. In a follow-up tweet, Almeida wrote: “The gains aren’t free: Jev can’t generate text.” He added that replacing sequential computation with parallel computation “is the same way Transformers leapfrogged RNNs.” The launch post lists the other limits: no image input “(yet…)”, cardinality capped at 255 for single-stage decisions, and string generation given up entirely. MindStudio’s launch explainer and RepoChad’s video add that model size, weights and local-deployment requirements are all undisclosed.

View original post on X →

This tells you what the “no type errors” guarantee is worth. The Hacker News launch thread (1,949 points, 511 comments) got there quickly. The top commenter, jacobgold, proposed a more accurate title: “Trading general purpose generation for fast typed inference”. The most-replied point in the thread was that Jev can’t emit an invalid type, but it can still emit a wrong valid value. Anthony Maio (September 16) put it most precisely: “Jev constrains the shape of the output. It does not constrain the judgment.” Gao Dalie used a railway image: the train is guaranteed to stay on the tracks, but nothing guarantees it picked the right destination. MarkTechPost is more direct: “‘Zero hallucinations’ means schema matching is guaranteed. The 0% figure is not empirical.” Coverage mostly skips the next point. Constrained decoding and structured outputs already give LLMs schema guarantees, so the type guarantee is a convenience and not the innovation.

View discussion on Hacker News →

View original post on Substack →

View original post on Substack →

“Is it just BERT?” The precise answer

The loudest technical debate has been whether Jev is a rebranded encoder classifier. The top comment (122 points) in r/LocalLLaMA’s “is it just a more generalised BERT?” thread answered: “It is BERT-like, but with the data, compute, and training recipe of modern LLMs.” In a Hacker News thread, Oras, who trained NLP models before LLMs, called it “just BERT with more data.” soerxpso replied that “Jev doesn’t require finetuning… The ability to knock out any arbitrary classification problem in minutes instead of in a week is a big deal.” Sebastian Raschka made the same point from the practitioner side (September 20). Encoder classifiers “were usually special-purpose and limited,” he wrote, while “the breakthrough of Jev is that it generalizes well,” and “the secret sauce is probably more in the data than in the training algorithm.”

View original post on Reddit →

View discussion on Hacker News →

View original post on X →

The distinction comes down to when the task gets defined. A BERT-style classifier fixes its task at training time. You fine-tune one model per task on labeled examples for a fixed label set, and that label set lives in the weights of the output head. Jev specifies its task at runtime. The question text and the allowed outputs are inputs, and one set of shared weights answers questions it was never specifically trained on. NLI-based zero-shot classifiers did a version of this back in 2019, so runtime task specification is not new in itself. The new part would be doing it at this breadth with probabilities that stay calibrated on tasks nobody trained for, and that second half is the part most coverage skips. The architecture debate stands in for the real question, which is about generalization plus calibration. Latent Space’s AINews had the right mental model: a “cheap, calibrated inference engine for structured choices.”

View discussion on Hacker News →

View original post on Reddit →

Why “output tokens free” is coherent and not a gimmick. In an LLM, the expensive part of a decision is decoding, because every output token needs another sequential pass. Jev’s output is a typed value plus at most 255 probabilities, so its compute scales with input. TypeSafe prices accordingly: $0.042 per million input tokens ($42 per billion), with output “FREE (too cheap to meter).” We ran a quick check on that price. XenoSpectrum paid $0.038 per 1,000 emails, which at TypeSafe’s rate works out to about 900 input tokens per call, consistent with one email plus a question. Mehdi Allahyari’s hands-on test (September 19) makes the same point from the LLM side: generating tokens for a categorization task “simply burns 30 extra forward passes.” In his test a single parallel pass was 3.2x faster, with 94% decision concordance, though on a small sample (n=50).

View original post on Substack →

To see the three primitives in use before you get access, Rob Shocks’ breakdown walks through the playground at 8:04, including how confidence is displayed.

The Calibration Backstory Nobody Drew

First, a definition. A model is calibrated when its stated probabilities match observed frequencies: among all the answers it gives at 0.9 confidence, about 90% are right. To check this, you draw a reliability diagram. Bin predictions by stated confidence, then plot each bin’s average confidence against its actual accuracy. A perfectly calibrated model sits on the diagonal. Points below the diagonal mean the model is overconfident. ECE is the weighted average gap between the curve and the diagonal. As elil17 put it in the HN launch thread: “If the value is 0.9 for 1000 answers, approximately 900 should be correct.”

Here is the point we think matters most, and no other coverage we have seen makes it. The GPT-4 Technical Report (OpenAI, March 2023) includes a calibration plot on a subset of MMLU. The pre-trained GPT-4 model was highly calibrated: its confidence closely tracked its probability of being correct. On the same subset, the post-trained model was not. In the report’s words, “post-training hurts calibration significantly.” For GPT-4, post-training centered on RLHF. OpenAI’s own chart, public for three and a half years, shows that optimizing for human preference measurably damaged the model’s sense of its own confidence.

Read Jev in that light. Almeida spent about four years on that post-training stack. In his AI Engineer talk (published July 31, 2026), he argued that RLHF made models “extraordinary at pleasing the human in the loop” and that “assistance and automation pull in different directions in optimization space.” On Hacker News, sothatsit made the technical version of the same point: “The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR. System One says they use RLCD… which presumably has accurate probabilities as an explicit optimisation goal.”

View discussion on Hacker News →

Our analysis: seen through the GPT-4 report, a GPT-4/RLHF veteran building a training method that optimizes calibration directly, instead of preference, is a coherent bet rather than a marketing flourish. You watch your field’s dominant post-training method break a property that automation depends on, and you build a method that targets that property. This is the strongest version of TypeSafe’s case. The connection is our reading of OpenAI’s report. TypeSafe has not, to our knowledge, framed it this way.

The same report also cuts against TypeSafe. The pre-trained base model was already well calibrated, which suggests you may not need exotic RL to get calibration so much as you need to avoid destroying it. That makes RLCD plausible, but it also means a cheap approach that reads option probabilities from a base model’s logits might recover much of the benefit. We come back to this in the Contrarian Corner.

Almeida’s own talk is worth your 18 minutes, since it sets out the RLHF-versus-automation thesis in his words. (The video’s title is the organizer’s framing. On the record, he worked on ChatGPT at OpenAI.)

RLCD vs RLHF: Outcomes vs Preferences

RLHF trains a reward model on human preference comparisons and then optimizes the policy to maximize that reward. The target is what people like, and people tend to like confident, fluent answers. Our argument is that this pressure pushes stated confidence up whether or not accuracy follows. According to TypeSafe’s launch post, Reinforcement Learning for Calibrated Decisions (RLCD) instead trains Jev to answer “with epistemically honest probabilities on System One tasks,” scored against outcomes. Viraj Phanse at Enterprise AI Weekly (September 21) summarized the contrast well: “RLHF asks, in some form, which response humans prefer.” Then he made the sharper point: “The interesting intellectual property in something like Jev… may not be the typed output at all. It may be whatever makes those probabilities trustworthy.”

View original post on Substack →

The mechanism that rewards honesty is a proper scoring rule. The two standard ones are log loss, the negative log of the probability you assigned to what actually happened, and the Brier score, the squared gap between your probability and the 0/1 outcome. Both are minimized in expectation only when you report your true belief. A worked example: suppose you believe an event is 70% likely.

You report	Expected Brier score (lower is better)	Expected log loss (lower is better)
50% (hedge)	0.250	0.693
70% (honest)	0.210	0.611
95% (inflate)	0.273	0.935

Hedging and inflating both lose. Nothing else in the scoring pushes the model toward honesty, which is why any method claiming calibration has to be anchored to one of these rules.

That raises the question in r/MachineLearning’s “How is RLCD RL?” thread. If Jev only outputs Choice, Score or Noul, the outputs are differentiable, and ordinary supervised training with log loss already targets calibration. So what does RL add? The top technical reply claimed Jev is “in reality an open source LLM with a few extra nn.Modules” and that its benchmarks treat “agreement with Astra/Fable” as success. Those claims are unverified, but if true they would matter. Being calibrated against another model’s answers is not the same as being calibrated against the world. The counterpoint came from ganzzahl: with PPO, the value model trained on a binary reward “is actually producing a probability.”

View original post on Reddit →

The synthetic-data question. Almeida said on September 17 that TypeSafe considers itself “a data research lab” and that “100% of our data is synthetic (but not the type of crap that is just spit out from an LLM obviously).” Keep in mind that calibration is not a fixed property of a model. It is a property of a model on a distribution. A model can be perfectly calibrated on the data it was trained on and badly overconfident on your support queue. That is the same problem we described in evals aren’t broken, eval users are: a synthetic user is not your user. So the real question is whether calibration learned on synthetic data transfers to production traffic. There is one encouraging signal. In jabr’s 947-case classifier benchmark, Jev lost only 1.0 point of accuracy under a distribution shift (0.974 to 0.964), while the open Von model lost 25.7. That is evidence of accuracy transfer, not calibration transfer. In the Kev thread on HN, hbarka asked the right structural question about open clones built on RLHF-trained bases: “how can the resulting model be considered Jev-like?”

View original post on X →

The Benchmark Spread Is the Story

This is the section to show your team. Every row below is a real measurement or a published claim. We sorted them by how independent the measurer is from TypeSafe.

Source	Who measured	Compared against	Speed advantage	Cost advantage	Independence
TypeSafe homepage figures, explained in the launch post	TypeSafe’s model-capabilities team, own workflows	Average of GPT-6 Astra and Fable 5.1	193.6x	444.6x	Vendor; self-described “higher end of real world gains,” “some bias could exist”
TypeSafe launch post (range)	TypeSafe	Frontier models (3–329 s vs Jev’s 70–500 ms)	40–200x	—	Vendor
Almeida launch tweet	TypeSafe CEO	LLMs	20–200x	40–400x	Vendor (different numbers from the blog)
Vercel, via TechCrunch	Pranit Sharma, production safety-review classifiers	ChatGPT Luna 5.6	5–18x (better accuracy)	—	Early-access developer
Bryo AI, via TechCrunch	CTO Nikhil Mudholkar	Gemini (slightly more accurate)	—	10–20x	Early-access developer
Interesting Engineering++	Roundup of independent measurements	Mistral Small 4	~5x	8.6x	Third-party roundup
XenoSpectrum / anisselbd	Independent author, 2,000 emails	Claude Haiku 4.5	~2.9x (239 vs 687 ms)	~12x ($0.038 vs $0.462 per 1k)	Independent, single run
jabr/classifier-benchmark	Independent, 947 cases	Local open encoders	Jev slower: ~330 ms via OpenRouter vs 46–73 ms	~$0.000014/call	Independent
r/LocalLLaMA Doom test	Community, 8 seeds	Laya, fine-tuned ModernCE, Qwen3.5-4B LoRA	Jev p50 117 ms; local encoders 7–15x faster	—	Independent (Jev won on play quality)

View original post on Substack →

View original post on Reddit →

Two things are true at once. First, the speed and cost advantages are real at every level of independence. None of the measurements we found show Jev slower or pricier than a frontier LLM on a decision task. Second, the size of the advantage shrinks sharply as the measurer gets more independent. On speed, going from 193.6x to Vercel’s 5–18x is a shrink of roughly 11–39x, and going to XenoSpectrum’s ~2.9x is roughly 67x. On cost, going from 444.6x to Bryo’s 10–20x is roughly 22–44x, and going to XenoSpectrum’s ~12x is roughly 37x. That is one to nearly two orders of magnitude between the headline and the most independent number. We explained how harness and setup choices move headline numbers this much in our look at eval harness economics. The vendor also isn’t consistent with itself: the blog says 40–200x faster, while the CEO’s tweet says 20–200x faster and 40–400x cheaper.

Some context the headline numbers leave out. XenoSpectrum measured a network floor of 163 ms for Jev versus 18 ms for Haiku. Subtracting each floor is a rough move, but it suggests a model-side gap of about 9x, with geography and network path eating much of the rest. TypeSafe itself concedes this in its caveat that its evals ran “from our laptops on the West Coast.” The baseline matters just as much. Compared with frontier LLMs, Jev is dramatically faster. Compared with a local encoder on your own GPU, it is slower. The 193.6x figure is averaged against GPT-6 Astra and Fable 5.1, which are reasoning models you should not be using as if-statements in the first place.

Headline numbers also spread faster than caveats. Bruce Burke’s “End of Autoregressive Monopolies” (September 21) repeats the “staggering 70 to 500 millisecond response window” and says the model “physically cannot invent a phantom tool,” without testing either claim. That is the bull case, and it is built entirely from the vendor’s best-case column.

View original post on Substack →

The Framing Problem

The most important practitioner finding so far is not a speed number. It is a swing in accuracy. anisselbd’s reproducible benchmark (September 17), written up by XenoSpectrum on September 20, ran PhishNChips v5.2, a set of 2,000 synthetic emails, with one call per email:

• Asked a single judgment question across all 2,000 emails, Jev scored 62.6%, against 81.3% for Claude Haiku 4.5 (no thinking).
• On 1,000 held-out emails, Jev asked a single signal question scored 89.4%. Decomposed into a five-question composite, it scored 95.0%, against Haiku’s composite at 93.2% and Haiku’s single-signal at 94.2%.
• A regex baseline scored 91.8%, within 3.2 points of Jev’s best result.
• Decomposition helped only Jev. The composite gap versus Haiku (95.0 vs 93.2) was not statistically significant (McNemar p = 0.063). The five questions were written after the author read the dataset’s own URL-evasion scheme. It was a single run by a single author, with no replication.

This matters because “just ask it” is the pitch and decomposition is the reality. On the same task, Jev ranged from roughly 19 points worse than a small LLM to at parity or slightly better, depending entirely on how the question was split up. Breaking a judgment into good sub-questions is prompt engineering, and here the sub-questions were tailored after studying the dataset. The cost of getting good answers has not gone away. It has moved from output parsing to question design, and you now own that work. The regex result is a reminder that for a large class of “decisions”, the best baseline is still twenty lines of rules.

The framing sensitivity reaches all the way down to option order. AVB (@neural_avb) reported on September 20: “Exact same prompts give you different probabilities when you run it multiple times. The ORDER of the choices DRASTICALLY changes the output probs.” If the stated probability of an option depends on whether it was listed first or third, that probability is not yet a stable measure of anything. Interesting Engineering++ also found that a forced three-option question with no valid answer still returned one of the three options. A typed model will always hand back a value of the right type, even when none of the allowed values is correct. Always give closed questions an escape option.

View original post on X →

The Promises Jev Has to Deliver

TypeSafe deserves real credit before the scorecard. Its launch post has a section headed “We love skeptics, and are skeptics ourselves.” It is the most candid launch disclosure we have read this year.

That candor is also why the gaps below are fair to name. Here are the seven promises the category claim depends on, with where the evidence stands today and what would settle each one.

1. Published calibration evidence. This is the load-bearing promise. Today: no reliability diagram, ECE or Brier score from TypeSafe, as MarkTechPost notes. AISeeKing’s explainer warns that Jev’s “confidence values should not automatically be interpreted as the probability that a decision is correct.” The community signals are small, unreplicated and point in different directions. A scam-job-ads benchmark on r/singularity measured ECE 0.046, which is good calibration, but “at 0.9 confidence it acts on 8 ads and catches 7 of the 53 scams. A calibrated score, and no usable threshold.” A September 21 r/MachineLearning post, citing jevals.com numbers, found larger calibration gaps against human labels for Jev than for small LLMs (yes/no: Jev 5.0 vs Gemini 3.8 Flash 2.0), but concluded “worse calibrated, better at knowing when it’s right.” Neither post has meaningful reach yet. Together they show why calibration has to be reported alongside resolution, meaning how often the model is confident enough to act on. Settled by: reliability diagrams and ECE per task class, plus coverage at each confidence level.

View the r/singularity scam-job-ads benchmark (Jev vs GLiNER2.5 vs a TF-IDF baseline) on Reddit →

View original post on Reddit →

2. Calibration that survives distribution shift. Today: the model is trained on 100% synthetic data, and jabr’s shift test covers accuracy only. Settled by: calibration curves on held-out, non-synthetic production datasets.

3. Robustness to question framing and option order. Today: XenoSpectrum found a 62.6% to 95.0% swing between framings, and AVB found order sensitivity. Settled by: variance across paraphrases and option permutations, published alongside accuracy.

4. Independent replication of speed and cost. Today: independent latency advantages range from about 2.9x to 18x against LLMs, and hosted Jev is slower than local encoders. Settled by: multi-region latency percentiles from third parties, with network floors reported separately.

5. Pricing sustainability. Today: TypeSafe itself says “We can’t prove it isn’t subsidized.” Settled by: price stability through general availability, or disclosed serving economics.

6. Beating the right baseline. Today: the headline compares against frontier LLMs. The classifiers Jev actually competes with are a different story. Merve Noyan of Hugging Face observed that people comparing Jev to GPT-5.6 have “never fine-tuned BERTForXYZ for living.” A HN reply reports 95% email accuracy from an embeddings-plus-logistic-regression model trained on “only 50-100 examples.” On scam ads, TF-IDF plus logistic regression hit F1 0.700 against Jev’s 0.312, and the regex above scored 91.8%. Jev does win some of these comparisons. On jabr’s benchmark it scored 0.965 against 0.59–0.70 for open zero-shot encoders and clones, and Von’s own README credits Jev with 96.6% against Von’s 72.0% across 49 tasks. Settled by: vendor benchmarks that include fine-tuned small encoders, zero-shot encoders and rules as baselines.

View original post on X →

View discussion on Hacker News →

7. Evidence beyond games and simulations. Today: MindStudio’s review of the demos covers Minecraft (about 1 cent per 2-minute session) and a drone sim (about 10 cents per 15 minutes), and notes that none were tested on real hardware, sensor noise or safety-critical failure modes. Even the game results disagree: the Doom test above had Jev winning, while Von’s README reports Von beating Jev in ViZDoom (9.00 vs 5.62 kills). Settled by: deployments with real sensors and real stakes, and published failure analyses.

What This Means for You (If You Build Agents)

Lucas Beyer’s deadpan reaction was the most-viewed skeptical reaction we found from launch week: “Very cool tech. It’s a complete mystery what kind of thing this can be used for though.” For agent builders the answer is not a mystery. TypeSafe’s own list of intended uses (smart if-statements, map-reduce over big data, real-time applications and “verify everything”) maps onto the decision points every agent loop already has.

View original post on X →

Where to try it now:

• Verification. Elvis Saravia built a custom verifier for the /goal feature in his agent harness that “checks whether the goal is actually complete.” That is the judge layer we have argued is becoming infrastructure, and a fast, calibrated judge is exactly what agent loops lack.
• Guardrails. Vercel’s Pranit Sharma used Jev for safety-review classifiers. Guardrails are the part of the guardrail stack teams keep rebuilding where both latency and cost compound.
• Routing and triage. The official typesafe-ai/skills repo ships the canonical example: “route incoming support tickets by department, with human review for uncertain decisions.”

View original post on X →

The escalation logic in that skills-repo example is where the value is. Patrick McGuinness explains why (September 18): “an overconfident wrong routing decision compounds through downstream agents before anything catches it. A decision layer that says I’m 43% sure and lets your code escalate accordingly solves a failure mode.” Nokast (September 17) makes the same case: “knowing when a model is uncertain can be as valuable as the model’s raw accuracy.” Bryo’s Mudholkar told TechCrunch that Jev “is the only one that hands back a real probability.” Armin Ronacher’s warning goes with that: “if this only comes back with 50% probability, maybe this is a coin toss.”

View original post on Substack →

View original post on Substack →

The real prize is a two-tier System 1 / System 2 agent. Jev decides when it is confident. The low-confidence band escalates to a frontier LLM, and from there to a human if needed. Jev does not replace System 2. As Theo put it, “it doesn’t replace reasoning models like Astra/Fable.” It decides when you need one. The open Verdict/OpenJev project already frames this as a four-tier ladder: deterministic code, then a System 1 decision model, then a generative LLM, then a human reviewer. It is the same multi-model routing pattern we recommend for provider-proof stacks, with a much cheaper first hop.

That pattern only works if calibration is real, which also makes it the test. Jev’s “0.9” is only useful as a threshold if 0.9 means about 90% on your traffic. XenoSpectrum’s advice is the right protocol: “count accuracy-by-confidence-bin yourself, on your own data.” In practice:

1. Shadow-run Jev next to your current decision path for one to two weeks. Log every prediction, its probability and the eventual correct answer.
2. Bin by confidence (deciles are fine) and compute accuracy per bin. That gives you your own reliability diagram.
3. Pick the threshold from your curve, not from the vendor’s claims. Then check coverage, the share of traffic above the threshold. A calibrated model that rarely reaches 0.9 gives you no usable threshold, which is what happened in the scam-ads test.
4. Re-check weekly for drift, and re-run whenever you change the question wording or the options.

The shape below follows the official SDK README as referenced in typesafe-ai/skills (client.system_one(state=..., questions={...})). Treat the field names as illustrative and confirm them against the SDK before use.

# Escalation gate: System 1 decides when confident, System 2 otherwise.
# Call shape per the official typesafe-sdk README; field names illustrative.
DEPARTMENTS = ["billing", "technical", "account", "sales", "none_of_these"]  # escape option

def route(ticket, threshold):
    res = client.system_one(
        state={"ticket": ticket.text, "customer_plan": ticket.plan},
        questions={"department": Choice(
            "Which department should handle this ticket?", options=DEPARTMENTS)},
    )
    label, p = res["department"].value, res["department"].probability
    shadow_log.append((ticket.id, label, p))          # you need this for the curve
    if p >= threshold and label != "none_of_these":
        return label                                   # System 1: act
    return escalate_to_llm(ticket)                     # System 2, then a human

# Calibration check on your own traffic: accuracy per confidence bin + ECE + coverage.
import numpy as np

def reliability(probs, correct, n_bins=10):
    probs, correct = np.asarray(probs), np.asarray(correct, dtype=float)
    bins = np.minimum((probs * n_bins).astype(int), n_bins - 1)
    ece, table = 0.0, []
    for b in range(n_bins):
        m = bins == b
        if not m.any():
            continue
        conf, acc = probs[m].mean(), correct[m].mean()
        ece += m.mean() * abs(acc - conf)
        table.append((f"{b/n_bins:.1f}-{(b+1)/n_bins:.1f}", round(conf, 3), round(acc, 3), int(m.sum())))
    return ece, table

def pick_threshold(probs, correct, target_acc=0.97):
    probs, correct = np.asarray(probs), np.asarray(correct, dtype=float)
    for t in np.arange(0.50, 1.00, 0.01):
        accepted = probs >= t
        if accepted.any() and correct[accepted].mean() >= target_acc:
            return round(t, 2), accepted.mean()   # (threshold, coverage)
    return None, 0.0                               # no usable threshold: resolution too low

Practical rules we would enforce on day one:

• Always include an escape option (“none / other / unsure”). Interesting Engineering++ showed that a typed model will otherwise force a valid-but-wrong answer.
• Test option-order stability. Shuffle the options, re-run the same inputs, and log the spread in probabilities. If a decision flips under reordering, that decision is not ready for automation.
• Count decomposition as prompt engineering you now own. Version the questions like code and re-validate calibration whenever they change.
• Benchmark against the right baseline. Use TypeSafe’s own adapter to compare against an LLM, and also train an embeddings-plus-logistic-regression model on 100 labeled examples and run your existing rules. If a regex gets within three points, keep the regex.
• Be skeptical of the ecosystem hype. The awesome-jev list already holds about 300 entries, and its maintainer warns: “Volume is not evidence of quality.”

What not to use it for yet: anything safety-critical, anything that needs generated text, and anything with more than 255 options in one stage. You can split a large label space into stages, but every stage adds its own calibration error.

Contrarian Corner: The Case That This Is Less New Than It Looks

View original post on X →

View original post on Reddit →

View original post on Reddit →

View discussion on Hacker News →

For the strongest version of the “it’s just a classifier” argument, with benchmarks against traditional classifiers, Prompt Engineering’s “JEV Is Just a Text Classifier” video is worth watching. Its central claim is that single-pass classifiers date to 2018 and zero-shot label pipelines to 2019.

Our Verdict

Jev is a genuinely good architectural idea. A typed, cheap, fast decision node is overdue in agent stacks, and TypeSafe is right that we have been using generation to make decisions and paying for it on every call. The speed and cost advantages hold up under independent measurement, even if they are one to nearly two orders of magnitude smaller than the headline.

The category claim is currently unproven. Without calibration, Jev is a very good fast classifier that competes with fine-tuned encoders, logprob tricks and regexes. With calibration, it becomes the missing component that lets an agent decide when to escalate. One piece of evidence would settle which of those it is: published reliability diagrams and ECE per task class, ideally on a held-out, non-synthetic distribution, with an ablation that separates RLCD from the architecture and the data. TypeSafe already runs a public evals dashboard with per-case walkthroughs. Adding reliability curves is a small step from there.

Our prediction is that the typed decision node becomes a standard slot in agent frameworks within the next two quarters, sitting between deterministic code and the frontier LLM, whether Jev fills that slot or an open encoder does. Whichever vendor publishes credible calibration curves first will own that slot. Our recommendation for builders is to start a shadow run this week on your highest-volume routing or verification step, build your own reliability diagram, and set a threshold only after your own data shows the probabilities are honest. Until then, treat Jev’s probabilities as a hypothesis to test, not a threshold to trust.

A team that writes “We love skeptics” into its launch post, ships the tooling to check its own claims and admits it can’t prove its pricing isn’t subsidized has earned a fair hearing. It has not yet earned trust in its probabilities. Publishing the calibration curves would change that.