How to read a model's calibration curve

A model is calibrated when its probabilities come true at the rate it states. If it gives a hundred picks a 60% chance, about sixty of them should win. If only fifty do, the model is overconfident. If seventy do, it is underconfident.

Calibration is separate from being right often. A model that gives every game 50% is calibrated if half its picks win, and useless. A model that picks winners well but states 80% when the truth is 60% is useful, and miscalibrated. You want both: good picks and honest numbers.

The curve groups graded picks into buckets by the confidence the model stated. Each dot is one bucket. Its position across the chart is the stated confidence, and its height is how often those picks actually won. The dashed diagonal is perfect calibration, where stated and realised rates match.

Dots below the line mean the model claimed more than it delivered in that range. Dots above it mean it delivered more than it claimed. A curve that sits below the line at high confidence and near it at low confidence is a common shape: the model is right about its ordinary picks and too sure about its strongest ones.

Each dot is a small sample of its own. A bucket of twenty picks that won 70% is consistent with a true rate well below or above that. Weigh dots by how many picks they hold, and distrust a striking dot built on a handful of results.

Calibration also drifts. A model can be well calibrated over a year and badly calibrated over the last month, as markets and seasons change. Look at the window as well as the shape.

Every strategy page on the Strategies board carries a calibration chart: each dot a confidence bucket for that strategy, the dashed line perfect calibration. We publish it whether it flatters the model or not. The Strategies table also shows a Brier score, a single-number summary of how close the stated probabilities came to the results.

The confidence shown on our picks is itself calibrated for display. We fit a curve from each model's graded history and use it to adjust the confidence a pick displays, with a cap, so a pick does not show a number the model's own record does not support. Calibration mostly changes the number you see. Selection runs on the model's raw output, with one exception: a pick whose calibrated confidence falls below a floor is not published.

Take a hypothetical strategy with four buckets. Picks it rated 50-55% won 53% of 400 picks. Picks rated 55-60% won 56% of 300. Picks rated 60-65% won 57% of 150. Picks rated 65-70% won 52% of 40.

The first two buckets sit close to the line on large samples: the model is honest in its ordinary range. The third sits a few points below on a fair sample: the model is somewhat overconfident there. The fourth looks bad, but forty picks is too few to say much. The shape to take from it is that confidence above about 60% has not been earned, so treat those picks as if they were in the 55-60% range.

A Brier score summarises calibration and accuracy in one number. For each pick, take the stated probability, subtract 1 if it won or 0 if it lost, and square the result. Average those squares across all picks. Lower is better; a model that says 50% on everything scores 0.25.

The Brier score is useful for comparing strategies at a glance, but it hides where a model goes wrong. Two strategies with the same score can fail in different ranges. Use the score to choose which curves to look at, then read the curves.

A well-calibrated model can still lose money. Calibration says the probabilities are honest. Profit also needs those probabilities to beat the price, and that is what closing line value and ROI measure. Read the curve next to those numbers, not instead of them.