Calibration — Llama Golf

Bet calibration · settled paper bets, excludes DG control · 9 portfolios in scope

Market: Matchup · Outright top-20 · Outright top-10 · Outright top-5 · Outright winner · All markets

What this means

Each dot is a probability bucket from the model's settled bets. The x-axis is what the model predicted; the y-axis is the empirical hit rate of bets in that bucket. A perfectly calibrated model lands on the diagonal — when it says 60%, it hits 60% of the time.

Dot size scales with sample count. The error bars are 95% Wilson confidence intervals on the empirical rate — wider bars mean the bucket has too few bets to be sure.

ECE (Expected Calibration Error) is the sample-weighted average gap between predicted and empirical across all bins. Lower is better. The Phase 3→4 gate calls for |Δ| ≤ 3pp across every bucket before deploying changes to the recommender.

Calibration matters more than accuracy. A model that says 50% and hits 50% of the time is honest signal — the betting math (Kelly, EV) assumes the probabilities are real, not just "high-confidence vs low-confidence."

Bin	n	Mean predicted	Empirical hit rate	95% CI	Δ (empirical − predicted)
40–50%	51	48.1%	43.1%	[30.5%, 56.7%]	-4.9pp
50–60%	69	51.9%	46.4%	[35.1%, 58.0%]	-5.6pp
Brier: 0.2481 ECE: 0.0529 Phase 3→4 gate: \|Δ\| ≤ 3pp across all buckets