Decision audit trail definition

A decision audit trail is the record that lets you reconstruct and review an automated decision later, which means storing the input exactly as the model saw it rather than as it looks today. Sanity stores content as typed fields with document revisions, so a decision log can point at the revision that was read, and replaying the decision reads that same state back instead of whatever an editor has changed since.

What goes into a decision audit trail?

A decision audit trail captures seven things, and dropping any one of them usually makes the record unreviewable.

First, the input state: the exact text or JSON the model read, pinned to a specific content revision rather than a pointer to the current version of the document. Second, the question or policy that was asked, with its version, because policies get reworded and a decision made under last quarter's wording is not the same decision. Third, the model version, since a hosted model can change behind a stable name. Fourth, the full output, meaning the probability assigned to every option in the answer space, not just the winning label. Fifth, the threshold in force at the time, for example "auto-approve at 0.95 or above, route to a human below that." Sixth, the action taken, which is what the system actually did with the result. Seventh, any human override, including who made it, when, and what they decided instead.

The fifth and sixth items are the ones teams most often skip, and they are the ones that answer the question a regulator or a customer actually asks, which is not "what did the model think" but "why did this happen to me."

Why does a decision audit trail matter for models that give no rationale?

A decision audit trail matters most for models that return no explanation, because it is then the only answer to "why did the system do that."

A growing class of models is built this way on purpose. TypeSafe AI released Jev on 15 September 2026 as what it calls a System One model: you pass a state, plus a typed question such as a Choice, a Score, or a Noul, and it returns probabilities over a fixed set of answers. There is no prose, no chain of reasoning, and no rationale to log. A generative model at least leaves behind a sentence you can argue with, even if that sentence is a post-hoc story rather than the real cause of the output. A decision model leaves behind a number.

That trade is often worth making, since a constrained output is easier to route on and cannot come back in a shape your code did not expect. But it moves the entire burden of explanation onto the record around the call. If the audit trail is thin, nobody can say anything defensible about a past decision beyond repeating the label it produced.

How do you investigate a decision when the model gives no reason?

You investigate a decision without a rationale by replaying it, not by asking the model to explain itself. Four techniques do most of the work.

Replay the stored state against the same model version and question version, and confirm you get the same distribution back. If you do not, something moved that you were not tracking, which is itself the finding.

Change one field in the state and replay. If removing a single sentence, a tag, or a reference flips the outcome, you have located the input the decision actually turned on. This is why the state needs to be structured rather than a flattened blob of text, since you cannot vary a field you cannot address.

Compare the decision against a set of examples a human has already labeled. A distribution of probabilities only tells you what the model does, not whether it is right, so agreement with reviewed labels is the closest thing to ground truth you will have.

Search the log for neighboring cases with similar states and different outcomes. Inconsistency across near-identical inputs is usually easier to act on than any single bad call.

Is a model's probability evidence about the person it was applied to?

No. A probability in a decision audit trail is a filter, not evidence about a person.

A score of 0.87 on a question like "is this submission spam" says the model sorted this item into a bucket with a certain confidence. It does not say the person who submitted it is 87 percent likely to be a spammer, and it does not carry the evidentiary weight of a human finding. Logs get read later by people who were not in the room, and a bare number in a record has a way of hardening into a fact about someone.

Write the audit trail so the distinction survives. Record the probability alongside the threshold it was measured against and the action it triggered, so the entry reads as "routed to manual review because 0.87 fell below the 0.95 auto-approve threshold" rather than as a verdict. Where a decision affects a person's access, money, or standing, the record should name the human who confirmed it, and the absence of that name should be visible in the log rather than implied.

A related point: vendor-published accuracy figures for a decision model are the vendor's own numbers unless someone independent has reproduced them. They belong in your evaluation notes, not in the audit trail as justification for a specific decision.

How does structured content make a decision audit trail reproducible?

Structured content makes a decision audit trail reproducible by giving the state a stable address and a bounded shape.

When the input to a decision is a typed field rather than a rendered page, you can log which field was read. When the categories the model chooses among are references to taxonomy documents rather than strings typed into a prompt, the answer space is defined in one place and versioned like everything else. And when every edit produces a document revision, the log can record the revision identifier, so a replay months later reads the content as it was at decision time rather than as it is now.

This is the architectural argument for running decision models against a content system rather than against scraped text. Sanity is the AI Content Operating System, and modeling content as typed fields, references, and revisions is what turns "the model looked at the product description" into a state you can fetch, diff, and re-run. Without that, the audit trail records a decision you can describe but cannot reconstruct, which in practice means you cannot defend it either.

Last updated: October 6, 2026