Calibration — Llama Golf

Bet calibration · settled paper bets, excludes DG control · 7 portfolios in scope

Market: Matchup · Outright top-20 · Outright top-10 · Outright top-5 · Outright winner · All markets

What this means

Each dot is a probability bucket from the model's settled bets. The x-axis is what the model predicted; the y-axis is the empirical hit rate of bets in that bucket. A perfectly calibrated model lands on the diagonal — when it says 60%, it hits 60% of the time.

Dot size scales with sample count. The error bars are 95% Wilson confidence intervals on the empirical rate — wider bars mean the bucket has too few bets to be sure.

ECE (Expected Calibration Error) is the sample-weighted average gap between predicted and empirical across all bins. Lower is better. The Phase 3→4 gate calls for |Δ| ≤ 3pp across every bucket before deploying changes to the recommender.

Calibration matters more than accuracy. A model that says 50% and hits 50% of the time is honest signal — the betting math (Kelly, EV) assumes the probabilities are real, not just "high-confidence vs low-confidence."

Bin	n	Mean predicted	Empirical hit rate	95% CI	Δ (empirical − predicted)
40–50%	42	48.0%	47.6%	[33.4%, 62.3%]	-0.4pp
50–60%	57	51.8%	43.9%	[31.8%, 56.7%]	-7.9pp
Brier: 0.2493 ECE: 0.0473 Phase 3→4 gate: \|Δ\| ≤ 3pp across all buckets