System One models in practice: a decision layer before the LLM.

In September 2026 TypeSafe introduced what it calls System One models, and the first of them, Jev. The pitch is narrow on purpose: unstructured state in, typed probabilistic decision out. No prose. The vendor quotes 70 to 500 milliseconds end to end and $0.042 per million input tokens, with output free.

The same week, we were building that exact shape of layer into Willit, a Finnish marketplace for local food that Viher IT builds and runs. Not because of the launch. Because of a bill.

The bill that started it

Willit has a background job that normalises demand signals: free text about what people want to buy, mapped onto a produce taxonomy. It called a small language model every five minutes. On 17 September it was retrying its way through a backlog of 1.58 million rows when the API credit ran out. Nothing dramatic broke. The job kept failing and reported the same error every five minutes for six hours.

So we inventoried every language-model call in the application. There were 26. A large share of them had the same form: a closed question with a fixed set of answers. What kind of message is this. Which of six business types is this company. Is this page a product page. Two of them ran on a mid-tier model to pick one label out of six. Eleven had no cost cap at all.

None of that is reasoning. It is classification, and we were paying generation prices for it.

What a System One model is

Jev gives up string generation entirely. You send it a state, which is a small data structure with text in it, and a set of questions. There are three kinds of question:

• Choice. Pick one option from a list. Returns the choice, the probability of every option, and a confidence.
• Score. Place the state on a rubric. Returns the level, the probabilities, and a confidence.
• Noul. Is this statement true. Returns a number between 0 and 1.

All the questions in one call are evaluated in parallel and in isolation, so adding a question barely changes the response time. That changes how you design. Instead of one clever prompt, you ask five plain questions at once.

This is what a support message looks like in TypeSafe's documented Python SDK:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

state = {"message": "Hei, tilasin perjantaina kalaa. Milloin sen voi noutaa?"}

with TypeSafeClient() as client:
    response = client.system_one(
        state=state,
        questions={
            "is_automated": Noul(
                instructions="Is `message` an automated reply or a notification?",
            ),
            "intent": Choice(
                instructions="What is `message` mainly about?",
                criteria={
                    "order_status": "Asks where an order is or when to collect it.",
                    "refund": "Wants money returned.",
                    "listing_help": "A seller asks how to list or edit a product.",
                    "other": "None of the above.",
                },
            ),
            "urgency": Score(
                instructions="How soon does `message` need a person?",
                criteria=["Can wait.", "Today.", "Now."],
            ),
        },
    )

intent = response.answers["intent"]
print(intent.choice, intent.confidence)

The useful idea is not the vendor. It is the shape: a closed answer set fixed in the request, probabilities back, and a threshold you control. Once your code speaks that shape, the model behind it is a replaceable part. That is how we built it.

The layer: cheapest tier first

Every inbound message passes through a cascade of three tiers before any expensive model is allowed to run.

• Tier 0, rules. Free, microseconds. A sender address that starts with no-reply is automated. A form tells you the lead type, because you know which form it was. A dictionary of produce names maps most demand text without any model. A rule is either certain or silent.
• Tier 1, a small classification model. The System One slot. Sub-second, and a few thousandths of a cent per decision.
• Tier 2, a small language model. Asked the same typed question and forced into the same closed answer set.

The rule that makes it pay: each tier receives only the questions the previous tier did not settle. When rules and the small model settle everything, no language model is called. That is the entire saving.

The types are small. This is a TypeScript version of what runs in Willit as Ruby:

type Question =
  | { key: string; kind: "choice"; ask: string; options: string[]; actAbove?: number }
  | { key: string; kind: "score"; ask: string; levels: string[]; actAbove?: number }
  | { key: string; kind: "yesno"; ask: string; actAbove?: number };

type Answer = {
  key: string;
  value: string; // always one of the listed options
  probabilities: Record<string, number>;
  confidence: number; // 0..1
};

interface Tier {
  name: string;
  answer(state: Record<string, string>, open: Question[]): Promise<Answer[]>;
}

const FLOOR = 0.6;
const ACT_ABOVE = 0.85;

const allowed = (q: Question): string[] =>
  q.kind === "choice"
    ? q.options
    : q.kind === "score"
      ? q.levels.map((_, i) => String(i))
      : ["true", "false"];

export async function decide(
  state: Record<string, string>,
  questions: Question[],
  tiers: Tier[], // cheapest first: rules, small model, small LLM
) {
  const settled = new Map<string, Answer & { tier: string }>();
  const suggested = new Map<string, Answer & { tier: string }>();
  let open = questions;

  for (const tier of tiers) {
    if (open.length === 0) break; // nothing left: no model is called

    // A tier never throws upward. Failure means "no answers from here".
    const answers = await tier.answer(state, open).catch(() => []);

    for (const a of answers) {
      const q = open.find((x) => x.key === a.key);
      if (!q || !allowed(q).includes(a.value)) continue; // closed set, enforced

      const found = { ...a, tier: tier.name };
      if (a.confidence >= (q.actAbove ?? ACT_ABOVE)) settled.set(a.key, found);
      else if (a.confidence >= FLOOR) suggested.set(a.key, found);
    }

    open = open.filter((q) => !settled.has(q.key));
  }

  // Whatever is still open goes to a person, best suggestion attached.
  return { settled, suggested, open };
}

Three details in that loop matter more than they look.

• The closed set is enforced on the way out, not trusted. An answer naming an option the question never listed is dropped. No free text leaves this layer, from any tier.
• A tier never throws upward. A failed tier returns nothing, and its questions fall through to the next tier or to a person, exactly as before the layer existed.
• A gate question can stop the cascade. If rules settle that a message is an out-of-office reply, nobody needs its intent. We found that one by building a simulator and watching the cascade ask anyway.

Confidence does the routing

Each question has two thresholds. Below the floor, the answer is ignored. Between the floor and the act threshold, it is a suggestion that the next tier or a person confirms. Above it, code may use it. Our defaults are 0.6 and 0.85, with 0.9 on the two questions where a wrong answer is expensive: urgency, and whether a message is automated.

Confidence needs one definition across tiers, or the thresholds mean nothing. For tiers that only return probabilities, we use the distance of the top option from chance:

// 0 when the top option is no better than chance, 1 when it is certain.
function confidence(probabilities: Record<string, number>): number {
  const p = Object.values(probabilities);
  const chance = 1 / p.length;
  return Math.max(0, (Math.max(...p) - chance) / (1 - chance));
}

Be honest about what that number is. A language model's self-reported probabilities are not calibrated. TypeSafe trains Jev's confidence separately but has published no reliability curve, so we treat it as unverified too. The thresholds are starting points to tune against a log, not facts.

What stays out of the model

TypeSafe's documentation is direct about the weak spots, and the design follows from them.

• Counting, dates and arithmetic stay in code. The model does not count and cannot judge whether a date falls inside a window. Quantities and deadlines are parsed by plain code.
• The state is short and relevant. Accuracy drops when the state carries material unrelated to the decision. Send the message, not the thread.
• The state is hostile. A small model does not defend itself against instructions hidden in its input. The closed answer set bounds the damage: the worst an injected message can do is mislabel itself.
• A classification is not permission to act. Knowing what a message is at 0.99 confidence changes nothing about what the system may do with it. In Willit that is decided by a separate autonomy ladder, and a confidently classified message still produces a draft until the action itself has earned trust.

A cheap tier has to earn its place

A cheaper model is only a saving if it is right. So nothing is switched over on belief.

1. Shadow. The tier answers and nothing uses the answer. The old path still decides, and its result is stored as the baseline.
2. Promote per question, not per set. A question goes live when at least 200 compared decisions show 97 % accuracy on the answers it would have acted on, and it would have acted on at least half. Rewording a question restarts its count.
3. Live, with a sample. The cheap answer is used when it is confident. The language model keeps running on a 5 % sample as a continuing baseline.
4. Demote automatically. If sampled agreement drops under 93 % over the last 100 decisions, the question goes back to shadow without anyone deciding it.

All of it hangs on one table: a decision log holding the answers, which tiers ran, the baseline, a human label where there is one, latency and cost. It stores a digest of the message, not the text. The rows a person has corrected become the golden set, and later the training data for a classifier of your own, if you ever want one.

One metric matters before promotion: of the answers the tier would have acted on, how many were right?

Failure needs a tier too

The credit incident taught the last part. A pipeline that calls a model has to tell a permanent error from a passing one.

• Circuit breaker. An authentication or billing error opens the breaker at once, for an hour, with one alert. Transient errors open it after three in a row, for five minutes. The old job had neither and raised the same alert 72 times.
• Cost guard. A daily budget per pipeline. The cheap tier is capped by call count rather than money, because a decision that costs two thousandths of a cent rounds to zero in a usage log.
• Kill switch. The whole layer sits behind one flag, and the vendor-backed tier behind another.

What it costs

The per-decision figures are our estimates for short support messages at list prices. The System One price is $0.042 per million input tokens, against roughly a dollar for a small language model, and there is no output charge. TypeSafe's launch post reports workflows running 193 times faster and 444 times cheaper, and says itself that those sit at the higher end of real-world gains.

Two things are worth more than the vendor's multiple. First, the biggest single saving in our audit was a dictionary: most demand text names a product a lookup table already knows, and then no model runs at all. Second, the layer pays for itself on rules plus a small language model alone. The System One tier is an upgrade to a design that already works, which is what lets us wait for evidence.

None of this is new as an idea. Model cascades were described in FrugalGPT in 2023. What is new is a model built for the cheap tier, with typed output as a guarantee rather than a hope.

What we do not trust yet

• "0 % hallucination" means type-safe, not true. The vendor says the figure is not empirical: the schema always matches, so the chart says zero. A wrong label from a closed set is still wrong.
• Nobody has measured Finnish. Our messages are in Finnish. We will replay a labelled set of real support cases through the cheap tiers, and that number decides, not a launch post.
• The processor is in the United States. A data processing agreement exists, and we have not signed one. Until that is settled the vendor tier stays closed, and the state is stripped of email addresses, phone numbers, identity numbers, bank accounts and links before it could go anywhere. Redaction is minimisation. It is not anonymisation, and it does not replace the agreement.
• Access is waitlisted. TypeSafe is opening early access gradually.

So here is the honest status. The layer is built and ships dark: off by default, shadow-only when on. We have no accuracy numbers yet, and this post does not pretend to. If the vendor tier never clears those checks, a self-hosted small encoder takes its slot and nothing else in the design moves.

Where to start

1. List every language-model call in your application. Mark the ones that are closed questions.
2. For each, ask what a rule could settle for free: sender, source, lookup table, regular expression.
3. Write the rest as typed questions: a key, a fixed answer set, a threshold.
4. Log every decision next to what the current path decided.
5. Run in shadow. Promote one question at a time, on evidence. Demote automatically.
6. Put a cost guard and a circuit breaker on every call site, including the ones you are not replacing.

The large model keeps the work it is good at: drafting, planning, and the case nobody anticipated. It just stops being the default answer to every question.

Sources

• TypeSafe (2026). Introducing System One models and Jev.
• TypeSafe documentation: primitives, confidence and patterns.
• Chen, L., Zaharia, M. & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.

Author

Risto HolappaFounder & Technical Lead, Viher IT Oy

Risto Holappa is the founder of Viher IT Oy and of Willit Oy, the local food marketplace this post draws on. He builds agentic software with classification services underneath, from Oulu, Finland.

LinkedInGitHub