Why so many LLM calls are really classification

A lot of production LLM traffic returns one word. Escalate this ticket or not. Spam or not spam. Which of six handlers gets this event. To get that word, you send the whole prompt to a general-purpose model, wait while it decodes a reply token by token, and parse a JSON object out of the result. You pay the price and the latency of text generation, and what comes back is a decision that fits in an enum.

“System One model” is TypeSafe AI’s name for a model built for that second shape: it takes a piece of state plus a set of typed questions and returns an answer to each one in a single pass, without generating text. The term borrows Daniel Kahneman’s split between fast, automatic “System 1” thinking and slow, deliberate “System 2” reasoning. Jev, TypeSafe’s implementation, is the worked example here. The label belongs to the vendor, but the idea behind it is older: if the output is limited to a known set of answers, the model can pick one instead of writing one. The question for your system is whether that shape matches a real bottleneck.

Why a System One model can skip token-by-token decoding

A generative LLM produces output it hasn’t seen yet. Each token depends on every token before it, so the model predicts one, appends it to the input, and predicts the next. That dependency is why generation is slow. Token 40 can’t start until token 39 exists.

A model of this kind removes the dependency by fixing the output space before the call. You don’t ask it to write an answer. For each question you give it a menu (pick one of these labels, place this on a 1-10 scale, answer yes or no), and it scores every option on every menu in one parallel forward pass. There is no sequence to decode, because every possible answer is known in advance.

That is the whole trick, and it is also the limit. Fixing the output is what lets you skip autoregression, and anything outside the fixed output is gone: you get no free-form text and no written-out reasoning to inspect. Jev’s own documentation says the model performs poorly on arithmetic, counting, and date comparison, which need multi-step computation rather than a choice between options. If the task needs the model to produce something rather than select something, use a different tool.

What you send a System One model and what you get back

Every call has two parts. State is whatever the model needs to look at: an email, a support ticket, a game frame, a row of application data. Questions are typed and bounded, using the three kinds above. For each question you get back the selected answer plus a probability distribution over the alternatives.

Build around that distribution. “Escalate” at 0.52 and “escalate” at 0.98 have the same label and mean very different things. The second you can act on automatically. The first belongs on a slower, more careful path, either a person or a full LLM call.

You can get a similar signal from a generative model by reading the token log-probabilities of a constrained answer, if your provider exposes them. The difference is that a System One model returns the distribution as the answer itself, for every question, without extra plumbing.

When to use a System One model in an LLM pipeline

The pattern that works is gating, not replacement. A fast typed model decides whether the expensive generative call needs to happen at all.

  • Routing. Classify the request before choosing which model or handler processes it, instead of asking one general model to classify and respond in the same call. A RAG pipeline can use the same gate to decide whether a question needs retrieval in the first place.
  • Moderation and triage. Score content for spam or urgency before it reaches a person or a more expensive model, so the expensive path only runs on what needs it.
  • Guardrails. After generation, check the response against typed constraints (does it contain a specific data pattern, does it fall into a banned category) as a cheap separate pass, instead of folding the check into the generation prompt.
  • Real-time control loops. A game loop or a robot controller that needs a decision every frame can’t wait for generation. A single-pass model fits inside that loop. A token-by-token model structurally doesn’t.

Things break when you treat the fast model’s answer as ground truth. Send low-confidence answers to the slower path, and set the threshold from your own labeled data, not from a round number that felt right. A flat distribution with a label on top is still a guess.

Alternatives: structured output and fine-tuned classifiers

Teams were solving this problem before the category had a name, in two main ways.

Structured output on a general-purpose LLM. The major LLM APIs can constrain generation to a JSON schema or an enum, which gives you the same typed-answer guarantee without adopting a new kind of model. Decoding is still autoregressive, but when the answer is a single enum value the generated part is only a handful of tokens. Much of the latency and cost goes into processing the prompt, and a System One model has to do that too. TypeSafe positions Jev as faster and cheaper than this approach. How much of that holds for your calls depends on your prompt length and how many questions you ask per call, so it’s a number to measure, not assume. Structured output also needs no new infrastructure if you already call that provider.

A small fine-tuned classifier. For one stable decision, like this exact escalation call or this exact spam filter, a classifier trained on your own labeled data is usually faster and cheaper than any general-purpose model, typed or not. The cost is that you need labeled data and a retraining run whenever the task drifts. A System One model’s edge over it is breadth: one model answers many different typed questions over different kinds of state, with no training run per decision.

The right choice depends on how many distinct decisions you have and how much traffic they get. If you have one stable, high-volume decision, use a dedicated classifier. If you have several related but varied decisions across a pipeline, that’s the case the System One shape is designed for. If you’re not making thousands of these calls a minute, start with structured output on the model you already call.

How to find the LLM calls that should be decisions

Go through what you already send to a general-purpose LLM and mark every call whose whole useful output could be an enum value or a number. Those are your candidates. Move two of them to structured output on a model you already use, either a hosted API or a local model running under Ollama, which accepts a JSON schema in its format parameter. Measure latency and cost per call against the current version.

Whatever you swap in, keep sending a few percent of the gated traffic through the old path too and compare the answers. If structured output already fits your latency budget and agrees with the old path, stop there. Evaluate a dedicated System One model like Jev only if it doesn’t, and use that side-by-side sample to judge it.