Is this true?
One probability, 0 to 1, that a proposition holds. There is no separate confidence field. The probability is the uncertainty.
noul("The message conveys urgency")
TypeSafe shipped a model that cannot write a single word and is still the right answer for about a dozen things we currently pay a frontier model to do. This is the research, the candidate list, and the one design rule I think we have to hold.
TypeSafe AI left stealth on 15 September 2026. Jev is their first "System One" model. It takes a state and a bag of named questions, and returns typed, calibrated answers. It cannot generate text, write code, call a tool, or remember anything. It decides.
One probability, 0 to 1, that a proposition holds. There is no separate confidence field. The probability is the uncertainty.
noul("The message conveys urgency")
One label out of up to 255, plus the probability of every option and a confidence number derived from how peaked that distribution is.
choice(instructions, criteria)
A position on a 2 to 10 level rubric. The answer can land between levels because it is probability weighted.
score(instructions, criteria)
| Endpoint | POST https://api.typesafe.ai/v1/systemone, bearer token, model jev-latest (today jev-1.13.0) |
|---|---|
| Latency | 70 to 500 ms |
| Price | $0.042 per million input tokens. Output tokens are free. |
| Limits | 64k tokens per request, 32k for state plus any one question, 255 choice options, 10 score levels, 1200 req/min, 250k tok/s |
| Client | npm i @typesafe-ai/sdk, Node 20+, typed inference on the answers object. Runs on Bun. |
| Shared weights | One model for every account. No fine-tuning, no LoRA. You steer it with state, instructions and criteria only. |
The property that changes how you design with it: every question in one request is evaluated in parallel and independently. Asking twelve questions costs roughly the latency of asking one. So you stop writing decision trees of sequential calls and start asking everything you might need up front, then branching in plain code.
import { experimental_evaluate as evaluate } from "ai"; const result = await evaluate({ model: "typesafe-ai/jev", state: { message: text, recentChannel: lastFive, agent: agentName }, questions: { addressed: { type: "boolean", instructions: "This message asks the assistant for something" }, urgency: { type: "score", instructions: "How time critical", criteria: ["whenever", "today", "right now"] }, kind: { type: "choice", instructions: "What is being asked", criteria: { question: "...", task: "...", chatter: "..." } }, }, }); result.answers.addressed.probability; // 0.94 result.answers.kind.choice; // "question" | "task" | "chatter", inferred from criteria result.providerMetadata?.typesafe?.confidence?.kind; // 0.88
We do not need a new dependency. We need an import. Vercel shipped evaluation as a first class modality in AI SDK 7. apps/core pins ai@7.0.107 and @ai-sdk/gateway@4.0.87, both already on disk. The API landed in 7.0.105.
I checked the installed .d.ts, not the docs. ai@7.0.107 exports evaluate as experimental_evaluate plus Experimental_EvaluationAnswer, Experimental_EvaluationModel, Experimental_EvaluationQuestion and Experimental_EvaluationResult. @ai-sdk/provider@4.0.17 carries the EvaluationModelV4 spec. @ai-sdk/gateway@4.0.87 exposes evaluationModel(modelId).
| Route | Cost to adopt | Trade |
|---|---|---|
evaluate({ model: "typesafe-ai/jev" }) over AI Gateway | zero new deps | Gateway billing and BYOK, but it cannot pin a model version today |
@ai-sdk/typesafe-ai@3.0.4 direct | one dep | Same evaluate call, TYPESAFE_AI_API_KEY, resolved version comes back |
@typesafe-ai/sdk, TypeSafe's own client | one dep, new shapes | No reason to, now that the AI SDK covers it |
The AI SDK renamed one question type. TypeSafe's noul is the AI SDK's boolean, and the field is probability, not noul. choice and score keep their names. Read TypeSafe's cookbooks for the thinking and the AI SDK types for the call.
The answer type is inferred from the criteria you pass, so a choice question narrows to a union of its own option keys. A typo in a branch fails the build.
type EvaluationAnswer<Q> = Q extends { type: 'choice'; criteria: infer C } ? { type: 'choice'; choice: Extract<keyof C, string>; probabilities?: Record<..., number> } : Q extends { type: 'score' } ? { type: 'score'; score: number; probabilities?: Record<string, number> } : { type: 'boolean'; probability: number };
EvaluationResult has answers, usage, warnings, rounding, providerMetadata and response. Confidence arrives under providerMetadata.typesafe.confidence[questionId], untyped. Every threshold in this document reads it, so that lookup needs one guarded helper rather than thirty optional chains.maxRetries defaults to 2. On a hot path with a 400 ms budget that is three attempts and a blown budget. Set it to 0 and let the caller fall through.abortSignal is a first class parameter. So the timeout is AbortSignal.timeout, no wrapper needed.rounding exists. Providers may round, and the SDK allows half a unit in the last place when it checks that distributions sum. Do not compare probabilities for exact equality.vercel/ai#21213, open. The Gateway returns the alias typesafe-ai/jev as modelId instead of the resolved version, and versioned ids such as typesafe-ai/jev-1.13 return 404 from the catalog. The direct TypeSafe provider returns the resolved version correctly.
That matters to us specifically. TypeSafe recommends pinning to avoid answer drift, and thresholds tuned against jev-1.13 are not thresholds for whatever ships next. My call: start on the Gateway because it is free to try, record response.modelId on every call anyway, and treat the move to @ai-sdk/typesafe-ai as the price of going to production with tuned thresholds.
TypeSafe publishes a jaggedness page per model version, which I respect. From jev-1.13:
P(noul) and P(not noul) need not sum to 1.The whole repo makes three model calls at runtime, and not one of them is a decision. Every classify, route, gate, rank and filter in Broods today is either the one big agent model reasoning inside its own tool loop, or a hand written regex. There is no small model path, no generateObject, no embeddings, no model router, no fallback chain.
| Call | Where | Shape | When |
|---|---|---|---|
streamText | harness/harness.ts:1829 | The agent loop, text plus tools | Every turn, multi step |
generateText | harness/compaction.ts:107 | Prose summary for compaction | Per turn when the char gate trips, blocking the first token |
transcribe | harness/transcribe.ts:75 | Audio to text | Per inbound audio attachment |
So the question is not "how do we add another model provider". It is "we have been making judgements with the two wrong tools, a regex and a 200 billion parameter poet, and there is now a third".
A Jev answer may narrow what the agent does. It may never grant something the agent did not already have.
This is the same invariant applyChannelRecord already holds in core, and it exists for the same reason. Jev does not treat its state as hostile, and our state is a Discord channel that anyone can type into. If a classifier can only subtract capability, a prompt injection in a channel message buys the attacker nothing.
Jev scores a bash command as dangerous, so a call that permissionMode: bypass would have run now asks the user. The regex already said yes. Jev said wait.
Jev scores a command as safe, so it skips an approval the regex wanted. Now a crafted command talks its way past the gate. Never do this.
The mention regex said no. Jev says the channel is clearly asking the on-call agent for help. Run. The blast radius is one extra agent run.
Someone typed @agent. That is an instruction from a human. A classifier does not get to overrule it, and a message that says "ignore the mention above" must not work.
Ordered by value over risk. Each one is independently shippable and independently revertible.
apps/core/src/harness/skills.ts:73 says it plainly: "Runs every turn, so every SKILL.md is read at once." The whole skill catalogue goes into the system prompt on every step, and the model burns a full extra model step calling load_skill before it can start the real work.
TypeSafe published a cookbook on exactly this, against a 182 skill roster, two stage: one request ranks the whole roster from truncated descriptions while gate questions ask whether any skill is needed at all, then a second request re-ranks the top 3 with full text and can still reject all of them.
| Measured | Baseline | With Jev | Oracle |
|---|---|---|---|
| Wrong skill loaded | 16.8% | 7.3% | 2.5% |
| Needless load on an uncovered request | 9.8% | 4.0% | 1.2% |
Their result comes with an honest footnote I want to keep: seven previously correct decisions were broken by a confident wrong suggestion. So the output is a hint, not a load. One line in the system prompt that the model is free to ignore.
Touches harness/session.ts:1365 (formatSkillsSystemPrompt) and nothing else. The load_skill tool stays exactly as it is.
prepareStep at harness/harness.ts:1004 already returns a PrepareStepResult and already runs every step. It refreshes instructions, leases and steering, and it never touches activeTools. Returning a narrowed tool list from a Jev choice over tool families is close to a one line hook.
This is the biggest win on accounts with several MCP servers attached, where every step pays for every tool schema. It is also the one where a wrong answer is most visible, because a tool the model wanted is suddenly missing. Start it in shadow mode: compute the narrowed set, record what it would have removed, change nothing.
harness/tools/filesystem-utils.ts:742 has a comment admitting the three classes its regex cannot catch: interpreter writes, symlinks, package manager prefixes. It is honest about being a heuristic, and it is the right heuristic to keep. Jev sits behind it.
A score question over the real command text, with a rubric from "reads only" to "destroys data outside the workspace", plus nouls for "writes outside the workspace" and "exfiltrates a credential". The result may only move a call from no approval to approval. Under the rule above, it can never do the reverse.
Adds roughly 100 to 300 ms to a bash call that was going to ask the user anyway, and nothing at all to one the regex already flagged, because you skip the call.
Today the agent runs only when literally mentioned. discord-channel.ts:817, slack-channel.ts:619, telegram-channel.ts:200, matrix-channel.ts:576, github-channel.ts:773 each implement their own regex version of the same question. A message that obviously needs the agent and forgets the @ is dropped into the context store and never answered.
The infrastructure for this already exists. Non triggering messages are already stored as context, so a classifier has the last few messages of the channel to read. One noul on 600 tokens of state costs about $0.000025.
This is the feature people will notice, and it is also the one I would not turn on by default. It changes what the product does, and it is where a prompt injection would land. Per channel record flag, off unless someone asks for it, and it may only ever add a run where the regex said no.
harness/questions.ts:78 maps a user's reply back onto the options the ask_questions tool offered, using /^\d+$/ and a lowercased exact string match. "the second one" and "yeah the postgres option" both fall through to free text. A choice over the options the agent itself just wrote is the textbook case, and the option set is ours, not the user's, so there is nothing for an injection to reach.
Subagent selection (session.ts:1382, run-subagent.tool.ts) is the same shape as skill selection and should reuse whatever we build for it, once that one has proven out.
The error classifiers (shared/channels.ts:286, transcribe.ts:115, sandbox/utils.ts:102) are regex maps from provider error text to a user facing category. They work. They break silently when a provider rewords a message. Low value, low risk, good first contribution for someone learning the codebase.
modelProviders.tsStill true, but for a smaller reason than I first wrote. modelProviderFactories() builds LanguageModel instances and the satisfies there fails the build when the 21 names drift. An evaluation model is a different spec, EvaluationModelV4, with its own resolver. It gets its own resolver next to that one, not a row inside it.
compaction.ts writes prose. Jev cannot write prose. It could pick a cheaper model or a better trigger than a 100k character count, but the summary itself needs a generative model. Separate problem, same file.
opa/broods_authz.rego is deterministic, auditable, deployed separately, and fails closed. Authorization does not get to be probabilistic. Jev advises before the gate. It is not the gate.
You asked whether Jev could be an agent. No. No memory, no tools, no generation, no loop. It is the part of a harness that decides, which is most of what an agent loop does between model calls, but it cannot be the loop.
Everything above makes the harness cheaper. This one gives the agent something it cannot do today.
The agent regularly faces "which of these 200 things matters". Grep returned 340 hits. A workspace has 80 files. An MCP server listed 60 tools. Right now the only way to answer is to read them all into context, which costs tokens, blows the window, and gets worse as the state grows.
A judge tool would let the model hand Jev a list and get back a calibrated probability per item in one 200 ms call. TypeSafe's own cookbooks already cover the shape: line by line semantic search, re-ranking, classifying RAG passages, per item noul when you need a count. 340 grep hits at 30 tokens each is about 10k tokens, which is $0.0004 and one round trip.
// the model calls this the way it calls grep judge({ items: grepHits, question: "Does this line actually implement the retry, rather than mention it?", }) // one evaluate() call, one boolean question per item, all answered in parallel // → [{ line: 88, p: 0.97 }, { line: 214, p: 0.04 }, ...]
That is a new public contract, so it drags the full walk: apps/docs/docs/api-reference/openapi.yaml, the docs, packages/demos, SDK types in packages/broods, and focused tests. Worth it, but after the harness side has proven the client is stable.
One module, one exported function, one config branch. Nothing else in the harness learns that Jev exists.
The config and client surface is worked out in its own artifact, Judge config, and that one is the current design. What follows is only the operational decisions, which have not changed.
| Concern | Decision |
|---|---|
| Dependency | None for the AI Gateway. ai@7.0.107 and @ai-sdk/gateway@4.0.87 are already in apps/core/package.json and already resolve experimental_evaluate. |
| Retries | maxRetries: 0. The SDK default of 2 turns a 400 ms budget into three attempts. |
| Timeout | abortSignal: AbortSignal.timeout(ms), fixed in core. Language model calls here deliberately run with { timeout: false }, which suits a model that streams for two minutes and not one that answers in 200 ms. |
| Failure | Return null, log, carry on. A classifier that adds latency when it is sick is worse than one that is absent. |
| Version drift | Record response.modelId on every call. The Gateway reads it back as the alias (#21213), so the day we need a real pin is the day we add @ai-sdk/typesafe-ai. |
| Observability | Reuse the span attributes from attemptRecordingMiddleware so judge latency sits beside model latency. Record the probability, the confidence and the threshold on every gated decision, or we cannot tune the thresholds later. |
| Tests | Experimental_EvaluationMockModelV4 from ai/test stubs the call, so thresholds are testable with no network. |
| Measurement | bench/ has a runner and recorded baselines. Every candidate lands in shadow mode first: compute the answer, record it, change nothing. Thresholds come from that data, not from the examples in their docs. |
Back of envelope, per agent run, against a $3 per million input token frontier model. The skill catalogue number assumes 50 skills at about 25 tokens of name and description each, re-sent on every step.
| Per run | Today | With Jev | Ratio |
|---|---|---|---|
| Skill catalogue in prompt, 6 steps | $0.0225 | $0.0004 | 56x |
The extra load_skill round trip | 2 to 6 s | 70 to 500 ms | ~10x |
| One wake decision, 600 tokens of state | n/a | $0.000025 | new |
| Relevance over 340 grep hits | $0.03 of context | $0.0004 | 75x |
The cost saving is real but it is not the reason to do this. The reason is latency and context. A judgement that used to cost a model step now costs 200 ms, which means we can afford to make judgements we currently skip entirely.
Broods, 21 September 2026. Revised after confirming AI SDK support against the installed types in apps/core/node_modules. Companion artifact: Judge config. Code references are line numbers on dev at 60375b20.