LLM evaluation (evals) definition

LLM evals are task-specific tests for an AI feature: a fixed set of inputs, the outcomes you consider correct, and a scoring rule, rerun every time the prompt, model, or data changes. The labels in that test set have to come from somewhere stable. In Sanity, a category set modeled as documents that fields point at by reference means the list of valid answers and the list of expected labels in your eval are one artifact, maintained in one place.

What is the difference between an eval and a benchmark?

A benchmark measures general model capability on a public, shared dataset so models can be ranked against each other, while an eval measures your system on your task with your data. MMLU, GSM8K, and HumanEval are benchmarks. A suite of 300 support tickets with the routing category a human assigned to each one is an eval.

The distinction matters because a model can top a public leaderboard and still fail your use case. The inputs you send may be distributed differently from the benchmark items, output can be sensitive to small prompt changes, and benchmark scores can be inflated by contamination, where test items leak into a model's training data. Researchers have argued that contamination has to be measured per benchmark rather than assumed away, in NLP Evaluation in Trouble.

The one-line version: a benchmark tells you which model is generally strong, and an eval tells you whether your feature works.

One wrinkle in how the word is used. "An eval" can mean a single scored test case or the entire suite, depending on who is talking. Both usages are common and you will see them in the same conversation.

What are the parts of an LLM evaluation suite?

An LLM evaluation suite has three parts: a test dataset, a scoring method, and a harness that runs them together.

The test dataset is a set of inputs paired with a reviewed definition of the correct response. When that set is curated, stable, and used as the baseline you compare every change against, it is usually called a golden dataset. Building one from imagination tends not to work. The reliable source of test cases is production failures you have already seen.

The scoring method splits cleanly in two. Anything expressible as logic should be checked with code: exact match against a label, regex, JSON parseability, schema validation, a tool call assertion. These cost almost nothing per result and give the same answer every time. Generated prose, where no single string is correct, needs rubric scoring, text-similarity measures such as BLEU or ROUGE, or a model grading the output. Hugging Face formalizes a suite as a set of (evaluator, dataset, metric) tuples, which is a useful way to see that all three have to be named before a score means anything.

The harness is the infrastructure that feeds each input through the system, records what happened at each step, grades the outputs, and aggregates per-sample scores into a number for the task. Teams run it like a regression test suite: on every prompt change, model change, or data change, with a threshold that gates the deploy.

What is the difference between offline and online evaluation?

Offline evaluation runs a curated dataset before you ship a change, and online evaluation scores live production traffic after you ship it. Both are LLM evals, and they catch different things.

Offline evaluation runs against test cases where you already know the expected outcome, so it catches regressions: a prompt edit that fixes one complaint and breaks ten cases that used to pass. Its weakness is that a fixed test set goes stale and stops resembling what users actually send.

Online evaluation runs continuously against production traces, usually on a sample rather than every request, using code checks, a model grading outputs, or user feedback signals. It catches drift, unexpected inputs, and failure modes nobody wrote a test for. Its weakness is the mirror image: there is no ground truth in production, so the signals are sparse and noisy. Langfuse's engineering guide sets the two out side by side in the form most practitioners now use.

The loop between them is the part worth internalizing. Offline evals validate a change before deploy, online evaluation surfaces the cases the dataset missed, and those cases get labeled and folded back into the dataset. That is how a golden dataset grows, and it is why the two practices are not alternatives.

How reliable is LLM-as-a-judge scoring?

LLM-as-a-judge means using a strong model to score another model's output against a written rubric. It exists because generated text has no single correct answer and human annotation does not scale, and the paper that popularized the technique found judge scores can track human preference closely enough to be useful, which is why the method spread.

The same paper documented the biases that come with it. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena names position bias, where the judge prefers whichever response it sees first, verbosity bias, where it prefers longer answers, and self-enhancement bias, where it prefers output from its own model family, along with weak performance on tasks requiring math or logic. Position bias has since been studied on its own in a systematic follow-up.

The practical consequence: a judge is itself a system under evaluation. It needs calibrating against human-labeled examples, it drifts when the underlying model is updated, and a judge score is evidence about quality rather than a measurement of it. Teams that treat judge output as ground truth end up optimizing for the judge.

Hallucination is a related but narrower case. Checking whether an output contradicts a source document is mechanically tractable, while checking whether it added unsupported claims is much harder, which is why faithfulness metrics are weaker than they look.

Why is evaluating a decision model easier than evaluating generated text?

Evaluating a decision model is easier because the output is a discrete value from a fixed answer space, so the correct answer is a label and scoring is exact match against that label. When you control the label set, no rubric is involved, no judge is involved, and therefore no judge bias, no judge drift, and no second model to calibrate. This holds for classification, routing, and scoring tasks, not for open-ended generation, where the problem is genuinely harder.

Classification metrics then apply directly, and they have been settled for decades: accuracy, precision, recall, F1, and the confusion matrix. You do not need new measurement theory to know whether a router works.

A second axis opens up that free-text evals cannot measure at all. If the model returns a probability alongside its answer, you can check calibration: whether things it called 90% likely happen about 90% of the time, using reliability diagrams, expected calibration error, and the Brier score. Calibration is what lets you set a confidence threshold and decide which cases run unattended. Keep in mind that calibration and accuracy come apart in both directions. A model can be accurate and badly calibrated, or perfectly calibrated and useless, like a predictor that always returns the base rate, so pair calibration with sharpness.

System One models are the current example of this shape. TypeSafe AI released Jev on 15 September 2026 as a model that takes state plus typed questions and returns probabilities over a fixed answer space with no prose, which means its outputs are scoreable against labels by construction.

What makes content structure part of an eval?

An LLM eval is only as meaningful as its inputs are stable, and content structure is what makes inputs stable. If two runs of the same test case feed the model different amounts of surrounding context, the scores are not comparable, and you have measured your prompt assembly rather than your model.

Sanity is the AI Content Operating System, the intelligent back end for companies building AI content operations at scale, and three things it does with content bear directly on evaluation. Content in the Content Lake is stored as typed fields and blocks, and a GROQ projection returns only the fields a question concerns, so the state in a test case is the specific field in dispute rather than a rendered page. A `reference` field names which document types it may point at, so when your categories live as documents, the valid answer space and the expected labels in your test dataset are the same list. And the History API returns a document as it was at a given revision or timestamp, so a test case can pin to the exact version it was labeled against, and a later edit to the live document does not quietly change what the eval is measuring.

That last point is the one teams discover late. Test cases built from live content drift with the content, and an eval whose inputs move is not a regression test.

Last updated: October 7, 2026