Leaderboard

This page is the cross-model scoreboard. Every row within a cohort uses the same eligible possessions, target, and game boundaries, with no information after the cohort's training cutoff. Training objectives remain model-specific. Bold values are best at the displayed precision. Lower error is better; higher skill is better.

This board measures in-season prediction after fitting target-season games. The Frozen Preseason Leaderboard separately holds out the entire regular season and playoffs with player values fixed before opening night.

Last generated: 2026-08-03 20:32 UTC from evaluation-2025-26-20260803T203258Z-84bd43a0.

Evaluation cohorts

The regular holdout is the final 186 regular-season games and is untouched by model selection. The playoff cohort contains all games in the playoffs partition and excludes play-in games. Every evaluated model is frozen before its cohort begins:

• regular holdout predictions use models fit on the first 1,044 regular-season games;
• playoff predictions use models refit on all 1,230 regular-season games;
• no playoff outcomes are used for fitting, calibration, or model selection.

Possession target

For eligible possession \(i\), define the offense-oriented point margin

Most possessions have zero defense points, but retaining the second term handles unusual opponent scoring without changing the target definition.

Only possessions with exactly one lineup segment are eligible. A possession with a substitution boundary is excluded in full rather than assigned to a starting or terminal lineup.

Possession metrics

For \(N\) eligible possessions:

RMSE and MAE are measured in points per possession. RMSE penalizes large errors more strongly; MAE gives the average absolute miss.

The mean reference predicts the offense-margin mean from the model's training window. Possession skill is

Positive skill beats the training mean; zero ties it; negative skill is worse.

Eligible-possession game margin

Let \(s_i=+1\) when the home team is on offense and \(s_i=-1\) when the away team is on offense. For game \(g\), aggregate only its eligible possessions:

Across \(G\) games:

This is deliberately named eligible-possession game-margin RMSE. It is not the error against the official final margin because points from excluded multi-lineup possessions are absent from both actual and predicted totals.

Using the corresponding mean-reference game predictions, game-margin skill is

Possession skill and game-margin skill therefore normalize improvement at different aggregation levels. Each skill score must be interpreted alongside the RMSE from the same level.

RAPM conversion

Ridge and Bayesian RAPM predict home net rating, while the common target is offense margin per possession. Their predictions are translated as

where \(\widehat{r}_i\) is the predicted home net rating for the possession's lineup and \(\overline{y}_{train}\) is the regular-training offense-margin mean. The factor 200 follows from comparing the same two lineups over two role-swapped possessions. If their signed lineup effect is \(\delta_i\), then

Regular-season holdout

Mean reference: possession RMSE 1.200988 and eligible-possession game-margin RMSE 17.4781.

Playoffs

Mean reference: possession RMSE 1.192560 and eligible-possession game-margin RMSE 16.7534.

Interpretation

Bayesian RAPM uses the same Gaussian prior and lambda corresponding to ridge, so its posterior mean is the ridge point estimate. Equal point-prediction metrics are expected. Bayesian value appears in uncertainty, interval calibration, and rank probabilities rather than lower posterior-mean RMSE.

The additive neural and Deep Sets exemplars select learning rate and AdamW weight decay by validation-possession-weighted MSE across expanding regular-season folds. CatBoost uses its resolved defaults and chooses its tree count from the latest chronological validation fold. RAPM + Transformer keeps the ridge prediction frozen and learns only a position-free attention residual, using a RAPM fit that excludes every validation or test game it predicts. Regular holdout and playoff outcomes remain outside every selection process.

Forward lagged-prior RAPM selects lambda independently within each historical season, carries the completed prior season's coefficient estimate forward, and uses zero for players without a prior-season estimate. Its regular-holdout state is fit only on the first 1,044 games; its playoff state is refit on all 1,230 regular-season games after selection.

Paired model comparisons

To preserve correlation among possessions from the same game, uncertainty is estimated by resampling complete games with replacement. Each row identifies its candidate and reference model. For bootstrap draw \(b\),

Negative differences favor the candidate. The interval is the 2.5th through 97.5th percentile of 2,000 paired game-cluster bootstrap draws. The final column is the share of draws where \(\Delta_b < 0\).

Correctness checks

tests/test_model_evaluation.py verifies offense-to-home aggregation, identical row counts and keys across models, playoff possession construction, metric calculations, and bolded-winner rendering. The evaluator additionally requires:

• validated source model manifests and exact artifact hashes;
• Bayesian and Transformer runs derived from the selected ridge run;
• matching regular-holdout game IDs across every model;
• matching possession, game, and player counts across neural-model sources;
• exact held-out possession keys for every stored prediction set;
• Bayesian and ridge posterior-mean equivalence within tolerance.

Run the focused checks with:

uv run pytest -q tests/test_model_evaluation.py

Reproduce

uv run nba-evaluate-models 2025-26 \
  --ridge-run-id baseline-2025-26-20260727T230533Z-72eac627 \
  --prior-rapm-run-id forward-lagged-rapm-2025-26-20260803T203054Z-c627d89d \
  --bayesian-run-id bayesian-2025-26-20260729T043953Z-b50cc2f7 \
  --neural-run-id neural-2025-26-20260729T173539Z-51bc0264 \
  --deep-sets-run-id deep-sets-2025-26-20260729T215128Z-dc12dd11 \
  --catboost-run-id catboost-2025-26-20260729T225755Z-9d5251ad \
  --rapm-transformer-run-id rapm-transformer-2025-26-20260729T233233Z-e316a73e

The underlying metrics.parquet, possession predictions, cohort summary, and source metadata are stored under artifacts/reports/model_evaluation/2025-26/evaluation-2025-26-20260803T203258Z-84bd43a0/.