Calibration — Llama Golf

Bet calibration · settled paper bets, excludes DG control · 14 portfolios in scope

Market: Matchup · Outright top-20 · Outright top-10 · Outright top-5 · Outright winner · All markets

What this means

Each dot is a probability bucket from the model's settled bets. The x-axis is what the model predicted; the y-axis is the empirical hit rate of bets in that bucket. A perfectly calibrated model lands on the diagonal — when it says 60%, it hits 60% of the time.

Dot size scales with sample count. The error bars are 95% Wilson confidence intervals on the empirical rate — wider bars mean the bucket has too few bets to be sure.

ECE (Expected Calibration Error) is the sample-weighted average gap between predicted and empirical across all bins. Lower is better. The Phase 3→4 gate calls for |Δ| ≤ 3pp across every bucket before deploying changes to the recommender.

Calibration matters more than accuracy. A model that says 50% and hits 50% of the time is honest signal — the betting math (Kelly, EV) assumes the probabilities are real, not just "high-confidence vs low-confidence."

Bin	n	Mean predicted	Empirical hit rate	95% CI	Δ (empirical − predicted)
40–50%	71	48.1%	45.1%	[34.0%, 56.6%]	-3.0pp
50–60%	102	52.1%	44.1%	[34.9%, 53.8%]	-8.0pp
Brier: 0.2496 ECE: 0.0596 Phase 3→4 gate: \|Δ\| ≤ 3pp across all buckets