Jev in Broods

TypeSafe shipped a model that cannot write a single word and is still the right answer for about a dozen things we currently pay a frontier model to do. This is the research, the candidate list, and the one design rule I think we have to hold.

What Jev actually is

TypeSafe AI left stealth on 15 September 2026. Jev is their first "System One" model. It takes a state and a bag of named questions, and returns typed, calibrated answers. It cannot generate text, write code, call a tool, or remember anything. It decides.

noul

Is this true?

One probability, 0 to 1, that a proposition holds. There is no separate confidence field. The probability is the uncertainty.

noul("The message conveys urgency")
choice

Which one of these?

One label out of up to 255, plus the probability of every option and a confidence number derived from how peaked that distribution is.

choice(instructions, criteria)
score

How much?

A position on a 2 to 10 level rubric. The answer can land between levels because it is probability weighted.

score(instructions, criteria)
EndpointPOST https://api.typesafe.ai/v1/systemone, bearer token, model jev-latest (today jev-1.13.0)
Latency70 to 500 ms
Price$0.042 per million input tokens. Output tokens are free.
Limits64k tokens per request, 32k for state plus any one question, 255 choice options, 10 score levels, 1200 req/min, 250k tok/s
Clientnpm i @typesafe-ai/sdk, Node 20+, typed inference on the answers object. Runs on Bun.
Shared weightsOne model for every account. No fine-tuning, no LoRA. You steer it with state, instructions and criteria only.

The property that changes how you design with it: every question in one request is evaluated in parallel and independently. Asking twelve questions costs roughly the latency of asking one. So you stop writing decision trees of sequential calls and start asking everything you might need up front, then branching in plain code.

import { experimental_evaluate as evaluate } from "ai";

const result = await evaluate({
  model: "typesafe-ai/jev",
  state: { message: text, recentChannel: lastFive, agent: agentName },
  questions: {
    addressed: { type: "boolean", instructions: "This message asks the assistant for something" },
    urgency:   { type: "score", instructions: "How time critical",
                 criteria: ["whenever", "today", "right now"] },
    kind:      { type: "choice", instructions: "What is being asked",
                 criteria: { question: "...", task: "...", chatter: "..." } },
  },
});

result.answers.addressed.probability;   // 0.94
result.answers.kind.choice;             // "question" | "task" | "chatter", inferred from criteria
result.providerMetadata?.typesafe?.confidence?.kind;  // 0.88

It is already in our tree

We do not need a new dependency. We need an import. Vercel shipped evaluation as a first class modality in AI SDK 7. apps/core pins ai@7.0.107 and @ai-sdk/gateway@4.0.87, both already on disk. The API landed in 7.0.105.

I checked the installed .d.ts, not the docs. ai@7.0.107 exports evaluate as experimental_evaluate plus Experimental_EvaluationAnswer, Experimental_EvaluationModel, Experimental_EvaluationQuestion and Experimental_EvaluationResult. @ai-sdk/provider@4.0.17 carries the EvaluationModelV4 spec. @ai-sdk/gateway@4.0.87 exposes evaluationModel(modelId).

Three ways in, and what each costs us

RouteCost to adoptTrade
evaluate({ model: "typesafe-ai/jev" }) over AI Gatewayzero new depsGateway billing and BYOK, but it cannot pin a model version today
@ai-sdk/typesafe-ai@3.0.4 directone depSame evaluate call, TYPESAFE_AI_API_KEY, resolved version comes back
@typesafe-ai/sdk, TypeSafe's own clientone dep, new shapesNo reason to, now that the AI SDK covers it

Naming, because it will trip someone

The AI SDK renamed one question type. TypeSafe's noul is the AI SDK's boolean, and the field is probability, not noul. choice and score keep their names. Read TypeSafe's cookbooks for the thinking and the AI SDK types for the call.

The types are better than I expected

The answer type is inferred from the criteria you pass, so a choice question narrows to a union of its own option keys. A typo in a branch fails the build.

type EvaluationAnswer<Q> = Q extends { type: 'choice'; criteria: infer C }
  ? { type: 'choice'; choice: Extract<keyof C, string>; probabilities?: Record<..., number> }
  : Q extends { type: 'score' }
  ? { type: 'score'; score: number; probabilities?: Record<string, number> }
  : { type: 'boolean'; probability: number };

Four things the types told me that the docs did not

The Gateway version pinning bug

vercel/ai#21213, open. The Gateway returns the alias typesafe-ai/jev as modelId instead of the resolved version, and versioned ids such as typesafe-ai/jev-1.13 return 404 from the catalog. The direct TypeSafe provider returns the resolved version correctly.

That matters to us specifically. TypeSafe recommends pinning to avoid answer drift, and thresholds tuned against jev-1.13 are not thresholds for whatever ships next. My call: start on the Gateway because it is free to try, record response.modelId on every call anyway, and treat the move to @ai-sdk/typesafe-ai as the price of going to production with tuned thresholds.

The honest limits

TypeSafe publishes a jaggedness page per model version, which I respect. From jev-1.13:

What I found in our codebase

The whole repo makes three model calls at runtime, and not one of them is a decision. Every classify, route, gate, rank and filter in Broods today is either the one big agent model reasoning inside its own tool loop, or a hand written regex. There is no small model path, no generateObject, no embeddings, no model router, no fallback chain.

CallWhereShapeWhen
streamTextharness/harness.ts:1829The agent loop, text plus toolsEvery turn, multi step
generateTextharness/compaction.ts:107Prose summary for compactionPer turn when the char gate trips, blocking the first token
transcribeharness/transcribe.ts:75Audio to textPer inbound audio attachment

So the question is not "how do we add another model provider". It is "we have been making judgements with the two wrong tools, a regex and a 200 billion parameter poet, and there is now a third".

Where judgement happens in a turn

regex or lookup table today the big agent model decides where Jev fits
INBOUND webhook gateway → core wake gate mention regex reach + invoke allow-list + OPA compaction big model, blocking system prompt assembly full skill catalogue + subagents + every tool schema MODEL LOOP pick a skill load_skill, costs a full step pick a tool no per-step narrowing pick a subagent run_subagent, up to 10 approve the tool call permissionMode + traversal regex + OPA WHERE JEV GOES skill suggestion rank all, verify top 3 one hint line in prompt −1 model step per turn tool narrowing prepareStep → activeTools choice over tool families smaller prompt per step semantic wake only when no mention noul on stored context answers without the @ danger scoring score the actual command, not the tool name catches interpreter writes and symlinks regex misses may raise approval, never lower it Every Jev box returns null on error or timeout and the current code path runs unchanged. Nothing above is on the critical path for correctness. Jev is advisory everywhere except where it tightens.

The one rule: Jev tightens, never loosens

A Jev answer may narrow what the agent does. It may never grant something the agent did not already have.

This is the same invariant applyChannelRecord already holds in core, and it exists for the same reason. Jev does not treat its state as hostile, and our state is a Discord channel that anyone can type into. If a classifier can only subtract capability, a prompt injection in a channel message buys the attacker nothing.

allowed raise the approval bar

Jev scores a bash command as dangerous, so a call that permissionMode: bypass would have run now asks the user. The regex already said yes. Jev said wait.

forbidden lower the approval bar

Jev scores a command as safe, so it skips an approval the regex wanted. Now a crafted command talks its way past the gate. Never do this.

allowed wake on an unmentioned message

The mention regex said no. Jev says the channel is clearly asking the on-call agent for help. Run. The blast radius is one extra agent run.

forbidden suppress an explicit mention

Someone typed @agent. That is an instruction from a human. A classifier does not get to overrule it, and a message that says "ignore the mention above" must not work.

Candidates, ranked

Ordered by value over risk. Each one is independently shippable and independently revertible.

1 Skill suggestion ship first contained

apps/core/src/harness/skills.ts:73 says it plainly: "Runs every turn, so every SKILL.md is read at once." The whole skill catalogue goes into the system prompt on every step, and the model burns a full extra model step calling load_skill before it can start the real work.

TypeSafe published a cookbook on exactly this, against a 182 skill roster, two stage: one request ranks the whole roster from truncated descriptions while gate questions ask whether any skill is needed at all, then a second request re-ranks the top 3 with full text and can still reject all of them.

MeasuredBaselineWith JevOracle
Wrong skill loaded16.8%7.3%2.5%
Needless load on an uncovered request9.8%4.0%1.2%

Their result comes with an honest footnote I want to keep: seven previously correct decisions were broken by a confident wrong suggestion. So the output is a hint, not a load. One line in the system prompt that the model is free to ignore.

Touches harness/session.ts:1365 (formatSkillsSystemPrompt) and nothing else. The load_skill tool stays exactly as it is.

2 Per step tool narrowing cheapest to try

prepareStep at harness/harness.ts:1004 already returns a PrepareStepResult and already runs every step. It refreshes instructions, leases and steering, and it never touches activeTools. Returning a narrowed tool list from a Jev choice over tool families is close to a one line hook.

This is the biggest win on accounts with several MCP servers attached, where every step pays for every tool schema. It is also the one where a wrong answer is most visible, because a tool the model wanted is suddenly missing. Start it in shadow mode: compute the narrowed set, record what it would have removed, change nothing.

3 Bash danger scoring real safety gain raise only

harness/tools/filesystem-utils.ts:742 has a comment admitting the three classes its regex cannot catch: interpreter writes, symlinks, package manager prefixes. It is honest about being a heuristic, and it is the right heuristic to keep. Jev sits behind it.

A score question over the real command text, with a rubric from "reads only" to "destroys data outside the workspace", plus nouls for "writes outside the workspace" and "exfiltrates a credential". The result may only move a call from no approval to approval. Under the rule above, it can never do the reverse.

Adds roughly 100 to 300 ms to a bash call that was going to ask the user anyway, and nothing at all to one the regex already flagged, because you skip the call.

4 Semantic wake gate biggest product change opt in per channel

Today the agent runs only when literally mentioned. discord-channel.ts:817, slack-channel.ts:619, telegram-channel.ts:200, matrix-channel.ts:576, github-channel.ts:773 each implement their own regex version of the same question. A message that obviously needs the agent and forgets the @ is dropped into the context store and never answered.

The infrastructure for this already exists. Non triggering messages are already stored as context, so a classifier has the last few messages of the channel to read. One noul on 600 tokens of state costs about $0.000025.

This is the feature people will notice, and it is also the one I would not turn on by default. It changes what the product does, and it is where a prompt injection would land. Per channel record flag, off unless someone asks for it, and it may only ever add a run where the regex said no.

5 Free text answer mapping small, obvious

harness/questions.ts:78 maps a user's reply back onto the options the ask_questions tool offered, using /^\d+$/ and a lowercased exact string match. "the second one" and "yeah the postgres option" both fall through to free text. A choice over the options the agent itself just wrote is the textbook case, and the option set is ours, not the user's, so there is nothing for an injection to reach.

6 Subagent routing and error classification later

Subagent selection (session.ts:1382, run-subagent.tool.ts) is the same shape as skill selection and should reuse whatever we build for it, once that one has proven out.

The error classifiers (shared/channels.ts:286, transcribe.ts:115, sandbox/utils.ts:102) are regex maps from provider error text to a user facing category. They work. They break silently when a provider rewords a message. Low value, low risk, good first contribution for someone learning the codebase.

Where Jev does not belong

Not in modelProviders.ts

Still true, but for a smaller reason than I first wrote. modelProviderFactories() builds LanguageModel instances and the satisfies there fails the build when the 21 names drift. An evaluation model is a different spec, EvaluationModelV4, with its own resolver. It gets its own resolver next to that one, not a row inside it.

Not replacing compaction

compaction.ts writes prose. Jev cannot write prose. It could pick a cheaper model or a better trigger than a 100k character count, but the summary itself needs a generative model. Separate problem, same file.

Not replacing OPA

opa/broods_authz.rego is deterministic, auditable, deployed separately, and fails closed. Authorization does not get to be probabilistic. Jev advises before the gate. It is not the gate.

Not an agent

You asked whether Jev could be an agent. No. No memory, no tools, no generation, no loop. It is the part of a harness that decides, which is most of what an agent loop does between model calls, but it cannot be the loop.

The bolder idea: Jev as a tool

Everything above makes the harness cheaper. This one gives the agent something it cannot do today.

The agent regularly faces "which of these 200 things matters". Grep returned 340 hits. A workspace has 80 files. An MCP server listed 60 tools. Right now the only way to answer is to read them all into context, which costs tokens, blows the window, and gets worse as the state grows.

A judge tool would let the model hand Jev a list and get back a calibrated probability per item in one 200 ms call. TypeSafe's own cookbooks already cover the shape: line by line semantic search, re-ranking, classifying RAG passages, per item noul when you need a count. 340 grep hits at 30 tokens each is about 10k tokens, which is $0.0004 and one round trip.

// the model calls this the way it calls grep
judge({
  items: grepHits,
  question: "Does this line actually implement the retry, rather than mention it?",
})
// one evaluate() call, one boolean question per item, all answered in parallel
// → [{ line: 88, p: 0.97 }, { line: 214, p: 0.04 }, ...]

That is a new public contract, so it drags the full walk: apps/docs/docs/api-reference/openapi.yaml, the docs, packages/demos, SDK types in packages/broods, and focused tests. Worth it, but after the harness side has proven the client is stable.

How it would be wired

One module, one exported function, one config branch. Nothing else in the harness learns that Jev exists.

The config and client surface is worked out in its own artifact, Judge config, and that one is the current design. What follows is only the operational decisions, which have not changed.

ConcernDecision
DependencyNone for the AI Gateway. ai@7.0.107 and @ai-sdk/gateway@4.0.87 are already in apps/core/package.json and already resolve experimental_evaluate.
RetriesmaxRetries: 0. The SDK default of 2 turns a 400 ms budget into three attempts.
TimeoutabortSignal: AbortSignal.timeout(ms), fixed in core. Language model calls here deliberately run with { timeout: false }, which suits a model that streams for two minutes and not one that answers in 200 ms.
FailureReturn null, log, carry on. A classifier that adds latency when it is sick is worse than one that is absent.
Version driftRecord response.modelId on every call. The Gateway reads it back as the alias (#21213), so the day we need a real pin is the day we add @ai-sdk/typesafe-ai.
ObservabilityReuse the span attributes from attemptRecordingMiddleware so judge latency sits beside model latency. Record the probability, the confidence and the threshold on every gated decision, or we cannot tune the thresholds later.
TestsExperimental_EvaluationMockModelV4 from ai/test stubs the call, so thresholds are testable with no network.
Measurementbench/ has a runner and recorded baselines. Every candidate lands in shadow mode first: compute the answer, record it, change nothing. Thresholds come from that data, not from the examples in their docs.

The money

Back of envelope, per agent run, against a $3 per million input token frontier model. The skill catalogue number assumes 50 skills at about 25 tokens of name and description each, re-sent on every step.

Per runTodayWith JevRatio
Skill catalogue in prompt, 6 steps$0.0225$0.000456x
The extra load_skill round trip2 to 6 s70 to 500 ms~10x
One wake decision, 600 tokens of staten/a$0.000025new
Relevance over 340 grep hits$0.03 of context$0.000475x

The cost saving is real but it is not the reason to do this. The reason is latency and context. A judgement that used to cost a model step now costs 200 ms, which means we can afford to make judgements we currently skip entirely.

Open questions for you

  1. Early access. Jev is early access. Do we have a key, and is the rate limit per account or per org? 1200 req/min sounds generous until one busy Slack workspace starts asking a wake question per message.
  2. Who pays. Account config carries provider keys today. Does Jev ride the same model, meaning each account brings its own key, or do we eat it centrally as part of the harness? At $0.042 per million with free output, eating it centrally is plausible and much simpler.
  3. Shadow first, or ship one? I would put skill suggestion straight into shadow mode behind a flag and let it run for a week against real traffic before it changes a single prompt.
  4. The wake gate. This is a product decision, not an engineering one. Do we want agents that answer when nobody tagged them?

Sources

Broods, 21 September 2026. Revised after confirming AI SDK support against the installed types in apps/core/node_modules. Companion artifact: Judge config. Code references are line numbers on dev at 60375b20.