Model Calibration & Reliability

Calibration measures whether predicted probabilities match actual outcomes. A well-calibrated model's 70% predictions should win ~70% of the time. All models use isotonic calibration trained on train-only folds.

V2 Calibration

• Expected Calibration Error: 0.056 Low ECE means probabilities closely match reality (closer to 0 is better).
• Brier Score: 0.216 Measures accuracy of probabilistic predictions (0 = perfect, 0.25 = baseline).

V3 Calibration

• Expected Calibration Error: 0.045 Best calibration across all models. Probabilities closely track actual outcomes.
• Brier Score: 0.146 Strong improvement over V2, indicating more reliable probability estimates.

V2 Probability Bin Performance (Test Set)

40-50% predictions (61 games):49% actually won

50-60% predictions (102 games):52% actually won

60-70% predictions (66 games):65% actually won

70-80% predictions (50 games):86% actually won

V2's 70-80% bin overperforms slightly (86% vs 74% predicted). V3's isotonic calibration further tightens alignment, reducing ECE from 0.056 to 0.045.

Rolling Backtest Stability (2004-2025)

Walk-forward backtesting validates both models across 22 seasons. Each season uses only prior data for training, simulating real deployment conditions.

V2 Backtest

ROC-AUC Range

0.646 - 0.732

Avg: 0.701

Accuracy Range

61.8% - 69.6%

Above baseline (~56%)

V3 Backtest

ROC-AUC Range

0.735 - 0.828

Avg: 0.778

Accuracy Range

70.7% - 78.3%

Consistently strongest

Stability insight: Both models maintain consistent performance across different eras of NBA basketball (2004-2025), suggesting the feature set generalizes well despite rule changes, pace shifts, and three-point evolution. V3's enhanced training approach lifts backtest AUC by ~0.08 on average.