Key takeaways
- The Brier score decomposes into reliability minus resolution plus uncertainty, so a perfectly calibrated model can still post a worse score than a miscalibrated one.
- In a 1,000-game counterexample, miscalibrated Model B scores 0.064 while calibrated Model C scores 0.068, because B's resolution on winners outweighs C's reliability.
- Resolution drives betting edge; calibration only matters once you use the probabilities for stake sizing like Kelly.
- Prefer uniform binning over sklearn's default quantile binning when auditing a reliability diagram, and check bin counts manually.
- Judge models with the Brier skill score against a base-rate baseline, not raw Brier scores, since uncertainty sets an unbeatable floor.
Brier score calibration alone can make a model worse: the score rewards sharpness (resolution) as much as calibration, so forcing probabilities toward base rates can inflate your Brier score while your edge stays flat. This guide decomposes the score into reliability, resolution, and uncertainty, reproduces a worked counterexample with hand arithmetic and Python, and shows how to audit a picks service's confidence numbers before you pay for them.
What does the Brier score actually measure beyond calibration?
The Brier score is the mean squared error of your probabilities: for each pick, square the gap between the stated confidence and the outcome (1 or 0), then average. Range is 0 to 1, lower is better, and it punishes two separate failures. Miscalibration: when you say 70%, you should win about 70% of the time. Low resolution: when you output timid probabilities near 0.5 on every game, you can be "calibrated" while telling a bettor nothing worth staking units on.
Murphy's partition of the probability score formalizes this as reliability minus resolution plus uncertainty, which is why a picks service's confidence scores need auditing on both axes, not one (our grading policy).
How do you decompose the Brier score into reliability, resolution, and uncertainty?
The Murphy decomposition splits the Brier score into three parts: Brier = REL − RES + UNC. Reliability punishes the gap between stated probabilities and observed hit rates in each bin. Resolution rewards bins whose hit rates differ from the base rate, which is the sharpness term. Uncertainty is the base rate's variance, ō(1−ō), and no model can move it.
Worked example: ten predictions, step-by-step arithmetic
Take ten picks in two bins: five at p = 0.8 (3 hit), five at p = 0.2 (1 hit). Base rate ō = 0.4.
| Bin | n | p | Hits | ō_k | n(p − ō_k)² | n(ō_k − ō)² |
|---|---|---|---|---|---|---|
| A | 5 | 0.8 | 3 | 0.6 | 5(0.2)² = 0.20 | 5(0.2)² = 0.20 |
| B | 5 | 0.2 | 1 | 0.2 | 5(0)² = 0.00 | 5(0.2)² = 0.20 |
- REL = (0.20 + 0.00) / 10 = 0.02
- RES = (0.20 + 0.20) / 10 = 0.04
- UNC = 0.4 × 0.6 = 0.24
- Brier = 0.02 − 0.04 + 0.24 = 0.22
Verify directly: bin A costs 3(0.2)² + 2(0.8)² = 1.40, bin B costs 1(0.8)² + 4(0.2)² = 0.80, and 2.20 / 10 = 0.22. The decomposition checks out. A service publishing calibrated confidence scores, like the audit trail at MVP Capping's auto-grading pipeline, should be able to hand you the per-bin numbers behind each term, not just the final score.
Why can a calibrated model have a worse Brier score than an uncalibrated one?
Calibration is necessary but not sufficient for a good Brier score (Brier, 1950). The score rewards resolution too, and a forecaster can be perfectly calibrated while resolving outcomes worse than a miscalibrated competitor. This is the counterexample worth internalizing before you audit any picks feed, including our own graded history at our autotrader record.
The counterexample setup
Take 1,000 games where 100 win (10% base rate). Model A always predicts 0.1: calibrated, zero resolution, Brier 0.09. Model B predicts 0.2 on the 100 winners and 0.0 on the 900 losers: miscalibrated on the winners, but it separates outcomes. Model C is calibrated with finer resolution than A, yet still loses to B.
Arithmetic for both models
Model B's Brier score is (100 × (0.2 − 1)² + 900 × 0²) / 1,000 = (100 × 0.64) / 1,000 = 0.064. Model C splits winners evenly: 50 predicted 1.0 (all correct, contributing 0) and 50 predicted 0.0 (all wrong, contributing 50). Each prediction group's observed frequency matches its prediction, so C is calibrated. It also splits losers: 450 predicted 0.0 (correct, 0) and 450 predicted 0.2 (all wrong, 450 × 0.04 = 18).
Total Brier = (50 + 18) / 1,000 = 0.068. Calibrated Model C loses to miscalibrated Model B, 0.068 to 0.064, because B's sharpness on the winners buys more than C's reliability earns. Pushing calibration alone, as most recalibration pipelines do, can move probabilities toward group means and destroy exactly the resolution that pays.
Runnable Python to verify the decomposition
import numpy as np
y = np.r_[np.ones(100), np.zeros(900)]
B = np.r_[np.full(100, 0.2), np.zeros(900)]
pC = np.r_[np.full(50, 1.0), np.full(50, 0.0), np.full(450, 0.0), np.full(450, 0.2)]
print(np.mean((B - y)**2)) # 0.064
print(np.mean((pC - y)**2)) # 0.068, calibrated but worse
When should you optimize calibration versus sharpness?
Resolution drives your edge. A model that separates 60% winners from 40% losers, even with sloppy probabilities, gives you bets worth taking and Kelly stakes worth sizing. Calibration only matters once you act on those probabilities: an uncalibrated 70% that fires at 55% real-world overbets your units and inflates bankroll variance.
So the priority depends on stage. Early, chase resolution: find probabilities that actually rank outcomes, because a perfectly calibrated but flat model beats nothing. Once resolution exists, calibrate before trusting the numbers for Kelly sizing, since miscalibration compounds through every stake.
Audit both on any service before paying. MVP Capping publishes auto-graded picks with confidence scores you can check against real outcomes (https://mvpcapping.com/es/autotrader/how-it-works), and CLV tracking tells you whether resolution is real or luck. A picks service with no outcome data offers neither, and no-vig market comparison won't fix that.
How do you compute and plot a reliability diagram in Python?
sklearn's calibration_curve gives you binned observed frequencies in two lines. Feed it outcomes (1/0) and predicted probabilities, and it returns the mean predicted probability and empirical win rate per bin, which you plot against the diagonal for a reliability diagram.
Manual binning vs sklearn calibration_curve
sklearn uses quantile binning by default (strategy='quantile'), which forces equal sample counts per bin. For a picks service audit, prefer strategy='uniform' plus a manual check of bin counts, because quantile bins can hide thin-sample bins where a model is badly miscalibrated. Manual binning with np.digitize lets you replicate the exact counterexample from the stats.stackexchange thread on Brier decomposition and verify the arithmetic yourself.
Bin count changes the picture: 5 bins smooths everything toward the diagonal, 20 bins exposes noise. At our Edge tier we publish the underlying probabilities precisely so you can re-bin and audit rather than trust our chosen binning.
How does Brier skill score relate to Brier score calibration?
The Brier skill score (BSS) normalizes the Brier score against a baseline, usually the climatology rate: BSS = 1 − (BS_model / BS_baseline). A BSS of 0 means the model is no better than always predicting the base rate; negative means it's worse. Unlike raw Brier, BSS is comparable across seasons and sports where base rates differ, so a 0.15 NFL and a 0.10 NBA BSS can't be read off raw scores alone.
This is why any picks service should publish BSS, not raw Brier: it answers the question you actually care about, does the model beat the market's no-vig implied probability? A well-calibrated model can still score BSS ≤ 0 if it never resolves outcomes better than climatology. Demand the baseline, not just the error.
Which calibration methods (Platt, isotonic, temperature scaling) change the Brier score and how?
Platt scaling fits a sigmoid to the model's logits, so it is monotonic: it can fix reliability but cannot reorder predictions, so resolution stays roughly intact. Isotonic regression fits a free-form step function and can flatten or sharpen any segment, which usually improves the reliability term more but risks overfitting resolution on small samples. Temperature scaling rescales logits by a single parameter, preserving ranking and, on neural nets, typically leaving resolution nearly unchanged while shifting reliability. Sample size is the practical divider. Platt and temperature scaling need few parameters and hold up on a few thousand outcomes; isotonic needs considerably more before its flexibility stops costing you resolution. All three are implemented in scikit-learn, whose CalibratedClassifierCV wraps any base model and whose calibration_curve bins predictions so you can plot the reliability diagram before and after.
For auditing a picks feed: if the service applies isotonic on a thin sample, expect good-looking reliability with degraded resolution, meaning calibrated but less sharp probabilities.
How should you audit a picks service's calibration before paying?
Demand the raw probability history, not a highlight reel. Every pick should come with its model probability, the market line at release, and the graded outcome. Bin those probabilities yourself into deciles, compute the empirical win rate per bin, and plot reliability. Then compute resolution: a service whose bins all sit near 50% regardless of the stated confidence has no edge, however well calibrated the curve looks.
Run the same math on any confidence scores before a subscription, and treat a service that deletes losing picks or refuses to export history as failing the audit outright.
Running the same audit on MVP Capping
We hold ourselves to the same standard: every pick is auto-graded, losses stay published, and CLV is tracked per pick so you can check calibration against closing lines, not just results. Where we fall short: our published history is shorter than older services like Sharp App or Action Network, so decile bins for low-probability ranges are thin, and you should weight resolution estimates accordingly until the sample grows.
FAQ
Is a lower Brier score always better for betting?
Lower is better only when comparing models that act on their probabilities the same way. A calibrated model can post a worse Brier score than a miscalibrated one, as the counterexample shows (0.068 vs 0.064), because the score rewards resolution as much as reliability. What matters for betting is whether the probabilities separate winners from losers, then whether they're trustworthy enough for sizing stakes.
What is a good Brier score for sports betting models?
It depends entirely on the base rate, since uncertainty, ō(1−ō), sets the floor no model can beat. A 10% base rate gives a floor of 0.09; a model scoring 0.064 beats it, one scoring 0.09 does nothing. Compare against the baseline with a Brier skill score instead: 0 means no better than always predicting the base rate, negative means worse.
Can calibration hurt resolution in small samples?
Yes. Isotonic regression, the most flexible recalibration method, fits a free-form step function and can flatten or sharpen any segment, which risks overfitting resolution when outcomes are scarce. Recalibration pipelines that push probabilities toward group means can destroy exactly the sharpness that pays. Platt scaling and temperature scaling use fewer parameters and preserve ranking, so they hold up better on a few thousand outcomes.
How many picks do I need before a reliability diagram means anything?
Enough that each bin has a meaningful sample, which depends on bin count. Five bins smooths everything toward the diagonal and can hide miscalibration; twenty bins exposes noise unless you have thousands of picks. Prefer uniform binning with a manual check of bin counts, and treat thin bins, especially in low-probability ranges, as unreliable until the sample grows.
Does the Brier score work for multi-outcome markets like 1X2 or props?
The standard formula handles binary outcomes, but multi-outcome extensions exist: you score each possible outcome with its stated probability and sum the squared errors, which stays in a comparable 0-to-1 range per event. The same reliability-versus-resolution tradeoff applies, so a calibrated 1X2 model can still lose to a sharper, miscalibrated one. Audit each outcome class separately.