Constrained decoding definition

Constrained decoding forces a language model to stay inside a defined output shape while it generates, masking out any token that would break the grammar or schema, so the response always parses. Most developers meet it as the structured output or JSON mode in a model API. It is a format guarantee, not a truth guarantee. Sanity, the AI Content Operating System, is where the schema on the other side of that guarantee usually lives: typed fields, references to taxonomy documents, and validation rules that decide what a valid answer even is.

How does constrained decoding actually work?

Constrained decoding works by filtering the model's vocabulary at each generation step. A language model produces a probability distribution over every token it could emit next. A constrained decoder takes that distribution, sets the probability of every token that would break the target format to zero, and then samples from what is left.

The filter comes from a grammar or a schema compiled into a state machine. If the output so far is `{"status": "`, and the schema says status must be one of three string values, the decoder only permits tokens that continue one of those three strings. Once a value is complete, the only legal next tokens are a closing quote, and whatever the schema allows after it. Backtracking is not needed, because an invalid token is never sampled in the first place.

This is why constrained decoding is sometimes described as having the parser drive the model, not the other way around. The model still chooses, but it chooses from a pre-filtered menu.

Where do developers encounter constrained decoding?

Developers meet constrained decoding in three common places. The first is the structured output mode built into commercial model APIs, where you attach a JSON Schema to a request and the provider guarantees the response conforms to it. The second is open source libraries such as Outlines, which compile a regular expression, JSON Schema, or context-free grammar into a token mask applied during sampling. The third is local inference runtimes, most visibly llama.cpp grammars, which accept a GBNF grammar file and restrict generation to it.

The practical payoff in all three is the same. Without constrained decoding, an application that expects JSON has to parse a response that might contain an apology, a markdown code fence, a trailing comma, or a truncated object, and then retry. With it, the parse step stops being a failure mode. Retry logic, repair prompts, and defensive parsing can be removed from the calling code.

Does constrained decoding stop hallucination?

No. Constrained decoding guarantees the shape of the answer, not the content of it. A model constrained to return one of five category labels will always return one of those five labels. Nothing in the mechanism makes it the right label.

This distinction became the center of a public argument in September 2026, after TypeSafe AI released Jev on 15 September 2026 and described it as a model that cannot hallucinate. Read carefully, the claim is about format: the output is constrained to schema-valid values. Critics pointed out, correctly, that constrained decoding already gives generative models schema-valid output, so a format guarantee on its own is not new. A confidently wrong but perfectly well-formed answer is still wrong, and it is arguably harder to catch, because every downstream validator passes it.

The useful mental model is not X, but Y: constrained decoding removes a class of parsing failures, but it removes nothing from the class of reasoning failures.

What is the difference between constrained decoding and a System One model?

The difference is not X, but Y: constrained decoding is a generative model made to obey a format constraint, while a System One model never generates tokens at all. With constrained decoding, the underlying model is still doing autoregressive generation, token by token, with a mask applied. With a System One model such as Jev, you pass a state (text or JSON) plus a typed question, and the model returns probabilities over a fixed answer space. There is no prose, no rationale, and no token stream to constrain.

Because both approaches produce schema-valid output, the format guarantee is not what separates them. The real differences are calibrated probabilities and cost per decision. A decision model returns a distribution over the allowed answers, so you can read how confident it is and route low-confidence cases to a human. Constrained generation gives you a sampled answer whose token probabilities are a poor proxy for confidence in the decision. A single forward pass over a fixed answer space is cheaper than generating a sequence of tokens, which matters when the same classification runs across a large content set.

The tradeoff runs the other way too. TypeSafe's performance and benchmark figures for Jev are the vendor's own and unreproduced, the model is closed and hosted with no open weights and no paper, and the absence of a rationale makes high-stakes decisions hard to audit after the fact. Injected text inside the state can still flip a decision, as it can with a constrained generative model.

What makes constrained decoding reliable in practice?

Constrained decoding is only as good as the schema you constrain to, which is why the design work sits in the answer space, not the prompt. Three things make it hold up.

First, a bounded answer space. A free-text string field constrained only to being a string gives you almost nothing. An enum of known values, or a reference to an existing taxonomy document, gives the decoder something real to enforce and gives you an answer you can join against other data.

Second, a clean state. The text or JSON you pass in determines what the model is deciding about. Unstructured input mixes the thing being judged with instructions, boilerplate, and anything a third party managed to write into the page, which is the opening prompt injection uses.

Third, an escape hatch. Every enum should include an explicit "unknown" or "needs review" value. Without one, a constrained model is forced to pick a wrong answer rather than decline, and the format guarantee hides the problem.

Structured content helps with all three. Typed fields, references to taxonomy documents, and document revisions turn content into a bounded state with a defined answer space, which is the input shape both constrained decoding and decision models were designed for.

Last updated: October 6, 2026