D40 / Territory 40 of 49

Model Validation and Backtesting

Determine whether an algorithm works out of sample.

10 released builds 1 algorithm familyIn release
D40 · MODEL VALIDATION AND BACKTEST…
D40Canonical / Tested / Open

Published work

Released builds
in this territory.

Every card below is included in the current governed release with its article, sources, Python, TypeScript, and available labs.

Search the released library
D40-F05

Classification and Score Validation

10 builds
01

Model Validation and Backtesting / Classification and Score Validation

ROC Curve and ROC-AUC

Build a tie-aware ROC ledger that exposes every threshold, weighted confusion count, curve point, and pairwise-ranking area.
10 min Python + TypeScriptAdvanced
02

Model Validation and Backtesting / Classification and Score Validation

Precision-Recall Curve and PR-AUC

Build a prevalence-aware precision-recall ledger and calculate average precision without silently substituting trapezoidal PR area.
10 min Python + TypeScriptAdvanced
03

Model Validation and Backtesting / Classification and Score Validation

Brier Score

Calculate a weighted Brier score with an auditable climatology baseline and no confusion between probability error and calibration alone.
10 min Python + TypeScriptIntermediate
04

Model Validation and Backtesting / Classification and Score Validation

Log Loss

Calculate binary log loss while making probability clipping, overconfidence, and weight semantics visible.
10 min Python + TypeScriptIntermediate
05

Model Validation and Backtesting / Classification and Score Validation

Reliability Diagram and Expected Calibration Error

Build a complete reliability table and equal-width ECE while preserving empty bins, support, and bin-sensitivity limitations.
10 min Python + TypeScriptAdvanced
06

Model Validation and Backtesting / Classification and Score Validation

Gains, Lift, and Decile Capture

Build a deterministic gains-and-lift table that exposes bucket assignment, ties, cumulative capture, and the random baseline.
10 min Python + TypeScriptAdvanced
07

Model Validation and Backtesting / Classification and Score Validation

Cost-Sensitive Threshold Optimization

Enumerate and audit every candidate threshold, confusion matrix, selected rate, and expected cost instead of hiding a policy inside one number.
10 min Python + TypeScriptAdvanced
08

Model Validation and Backtesting / Classification and Score Validation

Score Stability and Migration Matrix

Build a matched-entity stability and migration ledger with fixed bands, directional movement, and distribution change kept distinct.
10 min Python + TypeScriptAdvanced
09

Model Validation and Backtesting / Classification and Score Validation

Slice-Based Validation by Sector, Country, and Regime

Build a support-aware slice ledger that reports Brier, log loss, ROC-AUC, prevalence, and explicit low-support or single-class states.
10 min Python + TypeScriptAdvanced
10

Model Validation and Backtesting / Classification and Score Validation

Rare-Event Backtest and Confidence Bounds

Backtest a rare binary event with a point estimate, expected count, finite Wilson interval, and explicit sampling assumptions.
10 min Python + TypeScriptAdvanced

Territory coverage

What this domain
is mapped to cover.

The published builds above are the released slice of this map. Every subject below belongs to this territory's charter.

Walk-forward testingpurged cross-validationembargobootstrap confidence intervalsreality checkssensitivity analysisdrift and calibration tests