Brier score definition

A Brier score grades probability forecasts by squaring the gap between each prediction and what actually happened, then averaging. It rewards honesty, because a forecaster gets their best score by reporting what they really believe. It is used now to measure decision models, which answer typed questions with probabilities rather than text. Those probabilities are only as good as the state the model is shown, and in Sanity that state is a query against structured fields rather than a page of rendered prose.

How do you calculate a Brier score?

You calculate a Brier score by subtracting the outcome (1 if the event happened, 0 if it did not) from the probability you predicted, squaring that difference for every forecast, and taking the average. The formula for N binary forecasts is BS = (1/N) × Σ (f − o)², where f is the forecast probability and o is 1 or 0.

Here is a worked example with four predictions. You say 0.90 and it happens, so the squared error is (0.90 − 1)² = 0.01. You say 0.70 and it does not happen: (0.70 − 0)² = 0.49. You say 0.30 and it does not happen: (0.30 − 0)² = 0.09. You say 0.60 and it happens: (0.60 − 1)² = 0.16. The four errors sum to 0.75, and dividing by four gives a Brier score of 0.1875.

Two reference points make that number mean something. Always predicting 0.5 scores 0.25, so 0.1875 beats a forecaster who knows nothing, but not by a wide margin. And the single confident miss, 0.70 on something that did not happen, cost 0.49 of the 0.75 total on its own. A forecast of 0.70 that comes true costs only 0.09.

One caveat on range. The modern binary formulation runs from 0 to 1. Brier's original 1950 version sums the error across both classes and so runs from 0 to 2, giving exactly twice the value. Check which convention a number uses before comparing it to anything.

Why is a Brier score better than accuracy for probabilities?

A Brier score reads the probability itself, while accuracy throws the probability away. Accuracy picks a threshold, usually 0.5, converts every prediction to a yes or a no, and then counts how many were right. A model that says 95% and a model that says 51% score identically on accuracy whenever the answer turns out to be correct.

Take the same four predictions from the worked example above: 0.90 happened, 0.70 did not, 0.30 did not, and 0.60 happened. Thresholded at 0.5 they become yes, yes, no, yes against actual outcomes of yes, no, no, yes. Three out of four correct, so accuracy is 75%, which sounds respectable. The Brier score of 0.1875 tells you something accuracy cannot: the one mistake was made at 70% confidence, and that confident error is what dragged the score toward the know-nothing baseline of 0.25.

This gap matters wherever software consumes a probability rather than a label. If your code routes a document for human review when the model's confidence falls below 0.8, the exact number is the decision, and a metric that discards it is measuring the wrong thing.

Is the Brier score a proper scoring rule?

Yes. The Brier score is a strictly proper scoring rule, which means a forecaster minimizes their expected score only by reporting their true belief. Shading a prediction toward 0.5 to look cautious makes the expected score worse, and inflating it toward 1.0 to look decisive makes it worse too. The formal treatment is in Gneiting and Raftery's 2007 paper in JASA, which surveys strictly proper scoring rules and their estimation properties.

That property is the reason the Brier score survived from 1950 weather verification into machine learning evaluation. You cannot game it with a presentation strategy. The only way to improve is to know more about the question.

The other common strictly proper rule is log loss, also called the logarithmic score. Both punish being wrong, but they punish it differently. Brier is quadratic and bounded, so the worst any single binary forecast can cost is 1. Log loss is logarithmic and unbounded, so assigning a probability of 0 to something that then happens produces an infinite penalty. Put simply, Brier asks how far off you were, and log loss asks how badly you ruled out what actually happened. scikit-learn implements both, as brier_score_loss and log_loss.

What are the three components of a Brier score?

A Brier score decomposes into three parts: reliability, resolution, and uncertainty. The decomposition comes from Allan H. Murphy's 1973 paper "A New Vector Partition of the Probability Score"012%3C0595:ANVPOT%3E2.0.CO;2) in the Journal of Applied Meteorology, and it is written as BS = reliability − resolution + uncertainty.

Reliability is calibration in the narrow sense: of the things you called 70% likely, did about 70% of them happen? Lower is better. Resolution measures how far your forecasts move away from the base rate, which is why it is subtracted: a forecaster who always predicts the long-run average is perfectly reliable but useless. Uncertainty is a property of the questions, not the forecaster. It equals the base rate times one minus the base rate, and it captures how hard the question set is in the first place.

That third term carries the most important practical warning about the Brier score: two scores from different question sets are not comparable. A model scoring 0.08 on easy questions is not demonstrably better calibrated than one scoring 0.14 on hard ones. Comparison only means something when both forecasters answered the same questions.

Note also that the Brier score is designed for binary and nominal categorical outcomes. For ordered scales with three or more levels it treats being off by one level and off by five levels as equally wrong, which is why forecast verification generally uses the Ranked Probability Score, introduced by Epstein in 1969, for ordinal questions.

Why does the Brier score matter for decision models?

The Brier score matters for decision models because those models return a probability that software then thresholds, so calibration is as consequential as being right. A decision model answers a typed question about a piece of state with a number rather than generated text, and the number only carries information if 0.7 genuinely means something closer to seven-in-ten than to a vague gesture at likely.

Makers in this category have started publishing calibration alongside accuracy. Strands Labs, in its launch post for Strands Decider 2B, writes that it has been "measuring the first two together: accuracy on JevBench's public set, and calibration using the Brier score on the same set," with latency as the third target. Those figures are self-reported by the maker, as every accuracy and latency number in this young category currently is, and the benchmarks themselves differ between makers, so a Brier score from one model's evaluation set says nothing about another model's.

The input side deserves as much attention as the metric. A decision model scores a probability it formed from whatever state you handed it, so an incomplete or ambiguous state produces a confidently wrong answer that the Brier score will duly punish. Sanity, the AI Content Operating System, is the intelligent back end for companies building AI content operations at scale: content is modeled as typed fields and queried with GROQ, so the state passed to a model can be the two or three fields a question actually concerns rather than a scrape of a rendered page. Deciding what goes into the state is a design decision, and it happens upstream of any score you compute downstream.

Last updated: October 23, 2026