ROC Curve and ROC-AUC

Build a tie-aware ROC ledger that exposes every threshold, weighted confusion count, curve point, and pairwise-ranking area.

ROC Curve and ROC-AUC validation path from governed input to auditable diagnostic

Figure 1. Synthetic canonical path. Population, convention, intermediate evidence, output, and misuse boundary remain visible together.

The decision this tutorial makes visible

A single threshold hides how a score ranks records across operating points. ROC analysis makes that threshold sweep visible, but its apparent prevalence robustness does not make it a probability-calibration or cost metric.

The precise question is: Across every distinct score threshold, what false-positive and true-positive rates result, and what tie-aware area summarizes ranking discrimination?

A practitioner needs to know what the diagnostic does and does not justify. A builder needs a contract that can be reproduced from the same point-in-time inputs in Python, TypeScript, a visual, and a browser lab.

Intuition before notation

Move the threshold downward. Each tie group moves the operating point according to how much positive and negative weight it contains; the area summarizes the full path, not one deployment choice.

The result depends on the declared algorithm scope, input clocks, units, equality and rounding policies, and unsupported-state treatment. Change one of those and the output represents a different decision even when its field name is unchanged.

Scope and nearby methods

The canonical variant groups equal scores, starts at (FPR=0, TPR=0), updates once per distinct descending score, and integrates TPR over FPR by the trapezoid rule. At least one positive and one negative are required.

Variant	Definition	Best use	Main limitation
Canonical grouped empirical ROC	One point per distinct score and trapezoidal area	Auditable ranking validation	No uncertainty interval
Pairwise Mann-Whitney form	Weighted positive-negative comparisons with half credit for ties	Independent AUC check	Does not show operating points
Partial ROC-AUC	Area over a restricted FPR range	Low-FPR operating regions	Not comparable with full AUC

What is sourced, selected, synthetic, and derived

Role	Material claim	Evidence	Boundary
Sourced fact	ROC-AUC has a pairwise-ranking interpretation.	Hanley and McNeil (1982)	Not calibration or deployment utility.
Implementation choice	Equal scores update as one threshold group and trapezoids integrate the curve.	Frozen package contract	Other APIs may expose different threshold arrays.
Synthetic teaching input	The 24 records include deliberate score ties.	Repository fixture	Not production observations.
Author-derived calculation	Every point and area comes from weighted confusion counts.	Canonical fixture arithmetic	Conditional on eligibility and score direction.

The authoritative sources support only the exact facts named in the claim ledger. They do not certify the synthetic numbers in this tutorial. The repository fixture is deliberately invented for auditability, and the displayed output is author-derived under the selected implementation choice.

Formula, symbols, and numerical policy

Plain text

TPR(t)=TP(t)/P; FPR(t)=FP(t)/N; ROC_AUC=Σ (FPR_k-FPR_{k-1})(TPR_k+TPR_{k-1})/2

Symbol	Meaning	Unit	Policy
t	score threshold	score units	predict positive when score >= t
P,N	positive and negative weight	weight	both strictly positive
TPR	true-positive rate	fraction	TP/P
FPR	false-positive rate	fraction	FP/N
AUC	area under ROC	unitless	tie-group trapezoids

Use IEEE-754 binary64 arithmetic without intermediate rounding.
Group equal scores before ROC/PR curve updates unless a topic explicitly declares deterministic rank splitting.
Use positive finite weights; report counts and weight sums beside normalized metrics.
Round only for presentation and retain null for undefined diagnostics.

Read the formula in the same order as the algorithm. Validate identity, ordering, units, and supported state first. Apply the selected equality and window rules second. Calculate with unrounded numeric values. Round only at the declared presentation boundary, and preserve null as a diagnostic rather than coercing it to zero.

Build the algorithm

Validate identities, labels, weights, direction, and cutoff
Group records by descending score
Start at zero predicted positives
Add each tie group's positive and negative weight
Publish confusion rates after each group
Integrate adjacent points by trapezoids

Production-minded operational checklist

Freeze model version, population, score direction, and evaluation cutoff.
Verify outcome maturity and exclude future or revised evidence.
Reconcile record identities, weights, labels, and required slice or time keys.
Calculate the declared metric with visible intermediate denominators.
Review uncertainty and complementary diagnostics before any decision.

The checklist is intentionally strict: an explicit rejection is safer than a plausible output built from stale, malformed, or unsupported state.

Worked synthetic example

The canonical fixture is synthetic teaching data, not an observed control event, customer order, or broker execution. Its primary author-derived output, roc_auc, is 0.660714285714. The complete input and output are in datasets/canonical-input.json and datasets/expected-output.json.

Sort the synthetic records by descending score, update all equal-score records together, and accumulate weighted TPR and FPR. The output preserves the complete curve, class denominators, tie-group count, and area.

Counterfactual checkpoint

Reverse score quality. Move positive scores downward and negative scores upward while labels stay fixed. The output changes because pairwise ordering deteriorates

The structured result retains state and diagnostics in addition to the primary number. That makes the calculation independently reviewable and prevents a partial, null, rejected, or venue-bounded outcome from being mistaken for an unqualified value.

Boundary and counterexample workbook

The playground computes every scenario at 61 deterministic parameter states. The table uses the declared focus step and states whether that focus reproduces the canonical fixture. The full state ledger and compressed transition segments are in datasets/scenario-results.json.

Scenario	Review focus	Purpose	State	Primary output	Diagnostic	Decision segments
Canonical contract	Step 30 · canonical fixture	Exact canonical fixture at state 31; nearby states perturb one declared driver.	ranking-evaluated	ROC-AUC 0.660714	22 score groups; P=10.00, N=14.00	1
Stronger separation	Step 30 · comparison focus	Move positives and negatives apart while preserving labels and the evaluation cutoff.	ranking-evaluated	ROC-AUC 0.750000	21 score groups; P=10.00, N=14.00	1
Weaker or reversed separation	Step 30 · comparison focus	Compress and eventually invert score quality without changing outcome maturity.	ranking-evaluated	ROC-AUC 0.500000	1 score groups; P=10.00, N=14.00	1
Tie and boundary pressure	Step 30 · comparison focus	Quantize scores or probabilities to expose equality, bin, bucket, and threshold rules.	ranking-evaluated	ROC-AUC 0.632143	7 score groups; P=10.00, N=14.00	1
Prevalence and weight shift	Step 30 · comparison focus	Reweight event and non-event records while preserving identities.	ranking-evaluated	ROC-AUC 0.660714	22 score groups; P=14.00, N=19.60	1
Probability sharpness stress	Step 30 · canonical fixture	Move probabilities toward or away from endpoints while preserving score order.	ranking-evaluated	ROC-AUC 0.660714	22 score groups; P=10.00, N=14.00	1
Low-information comparison	Step 30 · comparison focus	Compress scores and probabilities toward the population center.	ranking-evaluated	ROC-AUC 0.660714	22 score groups; P=10.00, N=14.00	1

These rows are not backtest observations. They are controlled counterexamples that expose how one driver changes the state, output, or reason code while the rest of the contract stays fixed.

Visualize the boundary

ROC Curve and ROC-AUC annotated teaching map

ROC Curve and ROC-AUC calculation, rejection, and routing decisions

Decision takeaway: undefined, low-support, and rejected states remain visible rather than being coerced into zero.

Open this SVG at full size, or use the guided playground to compare the seven topic-specific canonical, boundary, policy, and failure scenarios.

The Mermaid flow answers where the selected calculation sits in the processing sequence. The SVG keeps the formula, output, decision boundary, and invariant visible together. The lab lets the reader step through the same structured states without changing the underlying definition.

Implementation walkthrough

The Python and TypeScript references begin with the same validation contract, reject malformed and unsupported state before calculation, preserve declared ordering and rounding policies, and return structured diagnostics rather than one context-free number.

The main implementation branches are:

Only one class is present — Reject ROC-AUC, because Both TPR and FPR denominators are required.
Scores tie — Update the complete tie group once, because Arbitrary within-tie order would invent curve detail.
Score direction is reversed — Declare and transform direction before evaluation, because Silent reversal changes AUC to a different question.

Neither reference silently fetches data, mutates caller-owned inputs outside the declared engine behavior, guesses hidden state, or substitutes a provider default. Shared JSON fixtures make value, null, state, and reason-code drift visible across languages.

Testing and validation

Definition tests compare every canonical field, reject malformed state, and exercise the material boundary. Family validation recomputes every playground state from the reference function. Independent arithmetic is recorded beside the fixture rather than inferred only from implementation output.

The audit must preserve these invariants:

population, positive-label orientation, weights, and cutoff
selected tie, integration, bin, bucket, threshold, band, slice, or interval convention
intermediate counts and denominators
primary metric and comparison baseline
undefined, rejected, low-support, and uncertainty diagnostics

Passing the suite proves selected-convention arithmetic and Python/TypeScript parity; it does not establish production fitness or an acceptance threshold.

Failure modes and misuse

A metric describes the declared evaluation population; distribution shift, label policy, interventions, and sampling can change its meaning.
One aggregate can hide threshold, calibration, segment, temporal, and uncertainty failures.
Passing implementation tests proves definition fidelity, not production reliability, fairness, legal compliance, profitability, or causal benefit.
Synthetic examples do not estimate real-world model performance or provide investment, lending, fraud, insurance, or regulatory advice.

Debugging order

When a result looks surprising, inspect the state in this order:

Confirm identifiers, scope, side, and decision clock.
Confirm units, ordering, and point-in-time inputs.
Confirm equality, rounding, null, and reset policies.
Recalculate the invariant and declared scenario focus before changing code.

Evidence and historical boundary

Historical decision: not useful. A named production model would add entity, privacy, label-maturity, sampling, policy, licensing, and causal-story risks without teaching the selected metric better than controlled synthetic records. The package therefore makes no claim about any real customer, issuer, fraud event, model approval, or future outcome.

The primary sources are Federal Reserve SR 26-2, Hanley and McNeil (1982), scikit-learn evaluation. They support the source roles listed in the research ledger, not a redistributable historical observation, a private participant decision, production conformance certification, execution-quality result, profitability claim, or prediction claim.

Choose the validation question first

These methods are complementary layers, not interchangeable scores.

Layer	Methods	Requires	Does not establish
Ranking discrimination	ROC-AUC · PR-AUC · Gains/Lift	scores + matured labels	Does not validate probability scale or choose a policy
Probability quality	Brier · Log Loss · Reliability/ECE	probabilities + matured outcomes	Does not replace ranking, costs, or support review
Decision policy	Cost-sensitive threshold	scores + labels + governed costs	The optimum changes with costs, prevalence, and constraints
Monitoring structure	Score migration · Slice validation	matched vintages or governed groups	Attrition, taxonomy, and support must remain visible
Sparse evidence	Rare-event confidence bounds	event count + trials + sampling model	A point estimate is incomplete without uncertainty

Five complementary classification-validation layers with ROC Curve and ROC-AUC highlighted

Method-selection takeaway: start from the decision question and available evidence. A strong rank does not prove calibrated probabilities; calibration does not choose a threshold; an aggregate does not prove slice or temporal stability.

Use the topic glossary to keep score, probability, label maturity, support, convention, and null diagnostics consistent across the family.

Use the guided learning lab

Question to answer: Separate one threshold operating point from the threshold-wide ranking summary, and see why tied scores move together.

Start with the canonical synthetic fixture and read the compact evidence trace.
Select the experiment that exposes the nearest boundary.
Change Separation stress and watch the topic-specific stage recompute.
Use Step and Back to connect the intermediate evidence to the primary diagnostic.
Compare the result with this guardrail: Both classes must be present, score direction must be declared, and ties cannot be split arbitrarily.

Open the standalone guided lab

Lab takeaway: ROC-AUC summarizes ordering across all grouped thresholds; the highlighted point is one possible operating policy, not the AUC itself.

Summary and next topic

You can now calculate and audit the selected classification-validation diagnostic. The learning flow is: Point-in-Time Data Test and time-aware validation → ROC Curve and ROC-AUC → Precision-Recall Curve and PR-AUC. Carry the result forward only with its scope, clock, state, and evidence label.

Workflows

Rendered from the canonical Mermaid sources linked by this article.

ROC Curve and ROC-AUC calculation flow

This flow identifies the selected calculation stages and the structured output.

Rendering system map…

Takeaway: ROC-AUC summarizes ordering across thresholds; it does not select a threshold or validate probabilities.

ReferencesPrimary sources and evidence notes

Expand the source trail, evidence role, and limitations behind the engineering choices.

S1 — Revised Guidance on Model Risk Management

Organization or authors: Board of Governors of the Federal Reserve System, OCC, and FDIC
Source type: Current interagency supervisory guidance
Publication or effective date: 2026-04-17
Version: SR 26-2
URL or DOI: https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm
Accessed: 2026-08-06
Jurisdiction: United States banking supervision
Supports: Validation and monitoring should assess reliability, limitations, performance deterioration, intended use, and data or model changes using a risk-based approach.
Limitations: Does not prescribe a universal metric, threshold, binning rule, or acceptance limit; applicability is supervisory and institution-specific.

S2 — The Meaning and Use of the Area under a Receiver Operating Characteristic Curve

Organization or authors: James A. Hanley and Barbara J. McNeil
Source type: Original peer-reviewed research
Publication or effective date: 1982-04-01
Version: Radiology 143(1), 29-36
URL or DOI: https://doi.org/10.1148/radiology.143.1.7063747
Accessed: 2026-08-06
Jurisdiction: Binary diagnostic ranking
Supports: ROC-AUC can be interpreted as the probability that a randomly selected positive receives a higher rating than a randomly selected negative, with ties requiring an explicit convention.
Limitations: The paper does not make ROC-AUC a calibration, cost, causal, or deployment-utility metric.

S3 — Metrics and scoring: quantifying the quality of predictions

Organization or authors: scikit-learn maintainers
Source type: Official maintained technical documentation
Publication or effective date: Current documentation accessed 2026-08-06
Version: scikit-learn 1.9 documentation
URL or DOI: https://scikit-learn.org/stable/modules/model_evaluation.html
Accessed: 2026-08-06
Jurisdiction: Software-library convention
Supports: Maintained definitions and API conventions for ROC-AUC, average precision, Brier loss, log loss, and classification metrics.
Limitations: Library behavior is not a regulatory standard and does not validate this repository's synthetic fixtures or selected governance policy.

Evidence boundary

Sources establish metric, statistical, and governance context. They do not verify the synthetic fixture, select a business threshold, or certify a deployed model.

Implementation

Full dependency-light reference implementations in both supported languages.

classification-validation.ts

/** Reference implementations for D40-F05 classification and score validation. */

type AnyRecord = Record<string, any>;

function numberValue(value: unknown, name: string): number {
  if (typeof value !== "number" || !Number.isFinite(value)) throw new TypeError(`${name} must be finite numeric`);
  return value;
}

function integerValue(value: unknown, name: string): number {
  const result = numberValue(value, name);
  if (!Number.isInteger(result)) throw new RangeError(`${name} must be an integer`);
  return result;
}

function records(inputs: AnyRecord, options: { score?: boolean; probability?: boolean } = {}): AnyRecord[] {
  if (!Array.isArray(inputs.records) || inputs.records.length === 0) throw new RangeError("records must be a nonempty array");
  const cutoff = inputs.evaluation_cutoff;
  if (cutoff !== undefined && (typeof cutoff !== "string" || cutoff.length === 0)) throw new TypeError("evaluation_cutoff must be a nonempty ISO-8601 string");
  const seen = new Set<string>();
  return inputs.records.map((raw: unknown, index: number) => {
    if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new TypeError(`records[${index}] must be an object`);
    const item = raw as AnyRecord;
    if (typeof item.id !== "string" || item.id.length === 0 || seen.has(item.id)) throw new RangeError("record ids must be unique nonempty strings");
    seen.add(item.id);
    if (item.label !== 0 && item.label !== 1) throw new RangeError("labels must be numeric 0 or 1");
    const weight = numberValue(item.weight ?? 1, `records[${index}].weight`);
    if (weight <= 0) throw new RangeError("weights must be positive");
    const row: AnyRecord = { ...item, label: item.label, weight };
    if (options.score) row.score = numberValue(item.score, `records[${index}].score`);
    if (options.probability) {
      row.probability = numberValue(item.probability, `records[${index}].probability`);
      if (row.probability < 0 || row.probability > 1) throw new RangeError("probabilities must be in [0,1]");
    }
    if (cutoff !== undefined) {
      for (const key of ["score_available_at", "label_available_at"]) {
        const timestamp = item[key];
        if (timestamp !== undefined) {
          if (typeof timestamp !== "string" || timestamp.length === 0) throw new TypeError(`${key} must be a nonempty ISO-8601 string`);
          if (timestamp > cutoff) throw new RangeError(`${key} exceeds evaluation_cutoff`);
        }
      }
    }
    return row;
  });
}

function classWeights(rows: AnyRecord[]): [number, number] {
  let positive = 0, negative = 0;
  for (const row of rows) row.label === 1 ? positive += row.weight : negative += row.weight;
  return [positive, negative];
}

function scoreGroups(rows: AnyRecord[]): Array<[number, AnyRecord[]]> {
  const ordered = [...rows].sort((a, b) => b.score - a.score || String(a.id).localeCompare(String(b.id)));
  const groups: Array<[number, AnyRecord[]]> = [];
  for (const row of ordered) {
    const last = groups.at(-1);
    if (!last || last[0] !== row.score) groups.push([row.score, [row]]);
    else last[1].push(row);
  }
  return groups;
}

export function rocCurveAuc(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { score: true });
  const [positive, negative] = classWeights(rows);
  if (positive <= 0 || negative <= 0) throw new RangeError("ROC requires positive and negative weight");
  let tp = 0, fp = 0;
  const points: AnyRecord[] = [{ threshold: null, true_positive: 0, false_positive: 0, true_negative: negative, false_negative: positive, tpr: 0, fpr: 0 }];
  for (const [threshold, group] of scoreGroups(rows)) {
    for (const row of group) row.label === 1 ? tp += row.weight : fp += row.weight;
    points.push({ threshold, true_positive: tp, false_positive: fp, true_negative: negative - fp, false_negative: positive - tp, tpr: tp / positive, fpr: fp / negative });
  }
  let auc = 0;
  for (let i = 1; i < points.length; i++) auc += (points[i].fpr - points[i - 1].fpr) * (points[i].tpr + points[i - 1].tpr) / 2;
  return { points, roc_auc: auc, positive_weight: positive, negative_weight: negative, tie_group_count: points.length - 1, state: "ranking-evaluated" };
}

export function precisionRecallAuc(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { score: true });
  const [positive, negative] = classWeights(rows);
  if (positive <= 0) throw new RangeError("precision-recall requires positive weight");
  let tp = 0, fp = 0, previousRecall = 0, averagePrecision = 0;
  const points: AnyRecord[] = [{ threshold: null, true_positive: 0, false_positive: 0, precision: 1, recall: 0 }];
  for (const [threshold, group] of scoreGroups(rows)) {
    for (const row of group) row.label === 1 ? tp += row.weight : fp += row.weight;
    const precision = tp / (tp + fp), recall = tp / positive;
    averagePrecision += (recall - previousRecall) * precision;
    previousRecall = recall;
    points.push({ threshold, true_positive: tp, false_positive: fp, precision, recall });
  }
  return { points, pr_auc_average_precision: averagePrecision, baseline_prevalence: positive / (positive + negative), positive_weight: positive, negative_weight: negative, tie_group_count: points.length - 1, integration_rule: "average-precision-right-step", state: "ranking-evaluated" };
}

export function brierScore(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { probability: true });
  const total = rows.reduce((sum, row) => sum + row.weight, 0);
  const eventWeight = rows.reduce((sum, row) => sum + row.weight * row.label, 0);
  const eventRate = eventWeight / total;
  const score = rows.reduce((sum, row) => sum + row.weight * (row.probability - row.label) ** 2, 0) / total;
  const baseline = eventRate * (1 - eventRate);
  return { brier_score: score, event_rate: eventRate, baseline_brier: baseline, brier_skill: baseline === 0 ? null : 1 - score / baseline, weight_sum: total, record_count: rows.length, state: "probability-evaluated" };
}

export function binaryLogLoss(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { probability: true });
  const epsilon = numberValue(inputs.epsilon ?? 1e-15, "epsilon");
  if (epsilon <= 0 || epsilon >= 0.5) throw new RangeError("epsilon must be in (0,0.5)");
  const total = rows.reduce((sum, row) => sum + row.weight, 0);
  const eventWeight = rows.reduce((sum, row) => sum + row.weight * row.label, 0);
  let loss = 0, clippedCount = 0;
  for (const row of rows) {
    const q = Math.min(Math.max(row.probability, epsilon), 1 - epsilon);
    if (q !== row.probability) clippedCount++;
    loss += row.weight * -Math.log(row.label === 1 ? q : 1 - q);
  }
  return { log_loss: loss / total, clipped_count: clippedCount, epsilon, weight_sum: total, event_rate: eventWeight / total, state: "probability-evaluated" };
}

export function reliabilityEce(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { probability: true });
  const binCount = integerValue(inputs.bins ?? 5, "bins");
  if (binCount < 2 || binCount > 20) throw new RangeError("bins must be between 2 and 20");
  if ((inputs.bin_strategy ?? "uniform") !== "uniform") throw new RangeError("only uniform bin_strategy is supported");
  const total = rows.reduce((sum, row) => sum + row.weight, 0);
  const ledgers: AnyRecord[][] = Array.from({ length: binCount }, () => []);
  for (const row of rows) ledgers[Math.min(Math.floor(row.probability * binCount), binCount - 1)].push(row);
  const bins: AnyRecord[] = [];
  let ece = 0, mce = 0, signed = 0;
  for (let index = 0; index < binCount; index++) {
    const members = ledgers[index], lower = index / binCount, upper = (index + 1) / binCount;
    if (members.length === 0) {
      bins.push({ index: index + 1, lower, upper, right_inclusive: index === binCount - 1, record_count: 0, weight_sum: 0, mean_probability: null, event_rate: null, signed_gap: null, absolute_gap: null });
      continue;
    }
    const weight = members.reduce((sum, row) => sum + row.weight, 0);
    const meanProbability = members.reduce((sum, row) => sum + row.weight * row.probability, 0) / weight;
    const eventRate = members.reduce((sum, row) => sum + row.weight * row.label, 0) / weight;
    const signedGap = eventRate - meanProbability, absoluteGap = Math.abs(signedGap);
    ece += weight / total * absoluteGap;
    signed += weight / total * signedGap;
    mce = Math.max(mce, absoluteGap);
    bins.push({ index: index + 1, lower, upper, right_inclusive: index === binCount - 1, record_count: members.length, weight_sum: weight, mean_probability: meanProbability, event_rate: eventRate, signed_gap: signedGap, absolute_gap: absoluteGap });
  }
  return { bins, expected_calibration_error: ece, maximum_calibration_error: mce, signed_calibration_error: signed, bin_count: binCount, weight_sum: total, state: "calibration-evaluated" };
}

export function gainsLift(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { score: true });
  const bucketCount = integerValue(inputs.buckets ?? 10, "buckets");
  if (bucketCount < 2 || bucketCount > rows.length) throw new RangeError("buckets must be between 2 and record count");
  const ordered = [...rows].sort((a, b) => b.score - a.score || String(a.id).localeCompare(String(b.id)));
  const totalWeight = ordered.reduce((sum, row) => sum + row.weight, 0);
  const totalEvents = ordered.reduce((sum, row) => sum + row.weight * row.label, 0);
  if (totalEvents <= 0) throw new RangeError("gains and lift require positive event weight");
  const prevalence = totalEvents / totalWeight;
  const members: Array<Array<[number, AnyRecord]>> = Array.from({ length: bucketCount }, () => []);
  ordered.forEach((row, rank) => members[Math.min(Math.floor(rank * bucketCount / ordered.length), bucketCount - 1)].push([rank + 1, row]));
  let cumulativeWeight = 0, cumulativeEvents = 0;
  const buckets = members.map((bucket, index) => {
    const weight = bucket.reduce((sum, [, row]) => sum + row.weight, 0);
    const events = bucket.reduce((sum, [, row]) => sum + row.weight * row.label, 0);
    cumulativeWeight += weight; cumulativeEvents += events;
    const populationShare = weight / totalWeight, eventCapture = events / totalEvents;
    const cumulativePopulation = cumulativeWeight / totalWeight, cumulativeGain = cumulativeEvents / totalEvents;
    return { bucket: index + 1, rank_start: bucket[0][0], rank_end: bucket.at(-1)![0], record_count: bucket.length, population_weight: weight, event_weight: events, population_share: populationShare, event_capture: eventCapture, bucket_lift: (events / weight) / prevalence, cumulative_population: cumulativePopulation, cumulative_gain: cumulativeGain, cumulative_lift: cumulativeGain / cumulativePopulation };
  });
  return { buckets, top_decile_capture: buckets[0].event_capture, overall_prevalence: prevalence, total_events: totalEvents, total_weight: totalWeight, bucket_count: bucketCount, tie_break: "score-descending-id-ascending", state: "ranking-evaluated" };
}

export function costSensitiveThreshold(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { score: true });
  if (!inputs.costs || typeof inputs.costs !== "object" || Array.isArray(inputs.costs)) throw new TypeError("costs must be an object");
  const names = ["false_positive", "false_negative", "true_positive", "true_negative"];
  const costs: AnyRecord = {};
  for (const name of names) costs[name] = numberValue(inputs.costs[name], `costs.${name}`);
  if (names.some(name => costs[name] < 0) || (costs.false_positive === 0 && costs.false_negative === 0)) throw new RangeError("costs must be nonnegative with a positive misclassification cost");
  const total = rows.reduce((sum, row) => sum + row.weight, 0);
  const thresholds: Array<number | null> = [null, ...[...new Set(rows.map(row => row.score))].sort((a, b) => b - a)];
  const candidates = thresholds.map(threshold => {
    let tp = 0, fp = 0, tn = 0, fn = 0, selected = 0;
    for (const row of rows) {
      const predicted = threshold !== null && row.score >= threshold;
      if (predicted) selected += row.weight;
      if (predicted && row.label === 1) tp += row.weight;
      else if (predicted && row.label === 0) fp += row.weight;
      else if (!predicted && row.label === 0) tn += row.weight;
      else fn += row.weight;
    }
    const expectedCost = (costs.false_positive * fp + costs.false_negative * fn + costs.true_positive * tp + costs.true_negative * tn) / total;
    return { threshold, true_positive: tp, false_positive: fp, true_negative: tn, false_negative: fn, selected_weight: selected, selected_rate: selected / total, expected_cost: expectedCost };
  });
  const optimal = [...candidates].sort((a, b) => a.expected_cost - b.expected_cost || a.selected_weight - b.selected_weight || -((a.threshold ?? Infinity) - (b.threshold ?? Infinity)))[0];
  return { candidates, optimal_threshold: optimal.threshold, optimal_expected_cost: optimal.expected_cost, optimal_selected_rate: optimal.selected_rate, optimal_confusion: { true_positive: optimal.true_positive, false_positive: optimal.false_positive, true_negative: optimal.true_negative, false_negative: optimal.false_negative }, costs, tie_break: "minimum-cost-then-lower-selected-weight-then-higher-threshold", state: "threshold-selected" };
}

function band(score: number, edges: number[]): number {
  let result = 0;
  while (result + 1 < edges.length && score >= edges[result + 1]) result++;
  return Math.min(result, edges.length - 2);
}

export function scoreMigration(inputs: AnyRecord): AnyRecord {
  if (!Array.isArray(inputs.baseline) || inputs.baseline.length === 0 || !Array.isArray(inputs.current) || inputs.current.length === 0) throw new RangeError("baseline and current must be nonempty arrays");
  if (inputs.higher_score_higher_risk !== true) throw new RangeError("canonical orientation requires higher_score_higher_risk=true");
  if (typeof inputs.baseline_observed_at !== "string" || typeof inputs.current_observed_at !== "string" || inputs.baseline_observed_at >= inputs.current_observed_at) throw new RangeError("baseline_observed_at must be before current_observed_at");
  if (!Array.isArray(inputs.band_edges) || inputs.band_edges.length < 3) throw new RangeError("band_edges must contain at least three values");
  const edges = inputs.band_edges.map((value: unknown) => numberValue(value, "band_edges"));
  if (edges[0] !== 0 || edges.at(-1) !== 1 || edges.slice(1).some((value: number, index: number) => edges[index] >= value)) throw new RangeError("band_edges must increase strictly from 0 to 1");
  function snapshot(rawRows: unknown[], name: string): Map<string, number> {
    const result = new Map<string, number>();
    rawRows.forEach((raw, index) => {
      if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new TypeError(`${name}[${index}] must be an object`);
      const row = raw as AnyRecord;
      if (typeof row.id !== "string" || row.id.length === 0 || result.has(row.id)) throw new RangeError(`${name} ids must be unique nonempty strings`);
      const score = numberValue(row.score, `${name}[${index}].score`);
      if (score < 0 || score > 1) throw new RangeError("migration scores must be in [0,1]");
      result.set(row.id, score);
    });
    return result;
  }
  const baseline = snapshot(inputs.baseline, "baseline"), current = snapshot(inputs.current, "current");
  if (baseline.size !== current.size || [...baseline.keys()].some(id => !current.has(id))) throw new RangeError("baseline and current must contain identical ids");
  const size = edges.length - 1, matrix = Array.from({ length: size }, () => Array(size).fill(0));
  const migrations: AnyRecord[] = [];
  let stable = 0, improved = 0, worsened = 0, absoluteMove = 0, scoreChange = 0, absoluteScoreChange = 0;
  for (const id of [...baseline.keys()].sort()) {
    const before = baseline.get(id)!, after = current.get(id)!;
    const a = band(before, edges), b = band(after, edges), move = b - a, delta = after - before;
    matrix[a][b]++;
    move === 0 ? stable++ : move < 0 ? improved++ : worsened++;
    absoluteMove += Math.abs(move); scoreChange += delta; absoluteScoreChange += Math.abs(delta);
    migrations.push({ id, baseline_score: before, current_score: after, baseline_band: a + 1, current_band: b + 1, band_move: move, score_change: delta });
  }
  const rowRates = matrix.map(row => { const total = row.reduce((sum, value) => sum + value, 0); return row.map(value => total === 0 ? null : value / total); });
  const baselineCounts = matrix.map(row => row.reduce((sum, value) => sum + value, 0));
  const currentCounts = Array.from({ length: size }, (_, column) => matrix.reduce((sum, row) => sum + row[column], 0));
  const n = baseline.size, baselineShares = baselineCounts.map(value => value / n), currentShares = currentCounts.map(value => value / n);
  const tv = 0.5 * baselineShares.reduce((sum, value, index) => sum + Math.abs(value - currentShares[index]), 0);
  return { band_edges: edges, matrix, row_rates: rowRates, baseline_band_shares: baselineShares, current_band_shares: currentShares, migrations, matched_count: n, stable_count: stable, improved_count: improved, worsened_count: worsened, stable_rate: stable / n, mean_band_move: migrations.reduce((sum, row) => sum + row.band_move, 0) / n, mean_absolute_band_move: absoluteMove / n, mean_score_change: scoreChange / n, mean_absolute_score_change: absoluteScoreChange / n, band_distribution_total_variation: tv, state: "migration-evaluated" };
}

function basicMetrics(rows: AnyRecord[], epsilon: number): AnyRecord {
  const total = rows.reduce((sum, row) => sum + row.weight, 0), [positive, negative] = classWeights(rows);
  const eventRate = positive / total;
  const brier = rows.reduce((sum, row) => sum + row.weight * (row.probability - row.label) ** 2, 0) / total;
  const logloss = rows.reduce((sum, row) => { const q = Math.min(Math.max(row.probability, epsilon), 1 - epsilon); return sum + row.weight * -Math.log(row.label === 1 ? q : 1 - q); }, 0) / total;
  const auc = positive > 0 && negative > 0 ? rocCurveAuc({ records: rows }).roc_auc : null;
  return { record_count: rows.length, weight_sum: total, positive_weight: positive, negative_weight: negative, event_rate: eventRate, brier_score: brier, log_loss: logloss, roc_auc: auc };
}

export function sliceValidation(inputs: AnyRecord): AnyRecord {
  const rows = records(inputs, { score: true, probability: true });
  if (!Array.isArray(inputs.slice_fields) || inputs.slice_fields.length === 0 || new Set(inputs.slice_fields).size !== inputs.slice_fields.length || inputs.slice_fields.some((field: unknown) => typeof field !== "string" || field.length === 0)) throw new RangeError("slice_fields must be unique nonempty strings");
  const fields = inputs.slice_fields as string[], minimum = integerValue(inputs.minimum_support ?? 4, "minimum_support");
  if (minimum < 2) throw new RangeError("minimum_support must be at least 2");
  const epsilon = numberValue(inputs.epsilon ?? 1e-15, "epsilon");
  if (epsilon <= 0 || epsilon >= 0.5) throw new RangeError("epsilon must be in (0,0.5)");
  for (const field of fields) if (rows.some(row => typeof row[field] !== "string" || row[field].length === 0)) throw new RangeError(`slice field ${field} is missing or invalid`);
  const overall = basicMetrics(rows, epsilon), slices: AnyRecord[] = [], eligibleBrier: number[] = [];
  let eligible = 0, flagged = 0;
  for (const field of fields) {
    const values = [...new Set(rows.map(row => row[field] as string))].sort();
    for (const value of values) {
      const members = rows.filter(row => row[field] === value), metrics = basicMetrics(members, epsilon);
      let status = "ok";
      if (members.length < minimum) status = "insufficient-support";
      else if (metrics.positive_weight <= 0 || metrics.negative_weight <= 0) status = "single-class";
      if (status === "ok") { eligible++; eligibleBrier.push(metrics.brier_score); } else flagged++;
      slices.push({ field, value, status, ...metrics, brier_gap_from_overall: metrics.brier_score - overall.brier_score, log_loss_gap_from_overall: metrics.log_loss - overall.log_loss, roc_auc_gap_from_overall: metrics.roc_auc === null ? null : metrics.roc_auc - overall.roc_auc });
    }
  }
  if (eligibleBrier.length === 0) throw new RangeError("no slice meets the canonical support and class requirements");
  return { overall, slices, slice_fields: fields, minimum_support: minimum, worst_slice_brier: Math.max(...eligibleBrier), eligible_slice_count: eligible, flagged_slice_count: flagged, state: "slices-evaluated" };
}

export function rareEventBounds(inputs: AnyRecord): AnyRecord {
  const events = integerValue(inputs.observed_events, "observed_events"), trials = integerValue(inputs.trials, "trials");
  if (trials <= 0 || events < 0 || events > trials) throw new RangeError("require 0 <= observed_events <= trials and trials > 0");
  const expected = numberValue(inputs.expected_probability, "expected_probability");
  if (expected < 0 || expected > 1) throw new RangeError("expected_probability must be in [0,1]");
  const confidence = numberValue(inputs.confidence_level, "confidence_level");
  const zMap = new Map([[0.90, 1.6448536269514722], [0.95, 1.959963984540054], [0.99, 2.5758293035489004]]);
  const z = zMap.get(confidence);
  if (z === undefined) throw new RangeError("confidence_level must be 0.90, 0.95, or 0.99");
  if (inputs.sampling_assumption !== "independent-bernoulli-approximation") throw new RangeError("unsupported sampling_assumption");
  const observed = events / trials, denominator = 1 + z * z / trials;
  const center = (observed + z * z / (2 * trials)) / denominator;
  const half = z * Math.sqrt(observed * (1 - observed) / trials + z * z / (4 * trials * trials)) / denominator;
  const lower = events === 0 ? 0 : Math.max(0, center - half), upper = events === trials ? 1 : Math.min(1, center + half);
  const consistency = expected < lower ? "expected-below-interval" : expected > upper ? "expected-above-interval" : "inside-interval";
  return { observed_events: events, trials, observed_rate: observed, expected_probability: expected, expected_count: trials * expected, confidence_level: confidence, z_value: z, wilson_center: center, wilson_half_width: half, wilson_lower: lower, wilson_upper: upper, consistency, sampling_assumption: "independent-bernoulli-approximation", state: "rare-event-evaluated" };
}

export function calculate(topicId: string, inputs: AnyRecord): AnyRecord {
  if (!inputs || typeof inputs !== "object" || Array.isArray(inputs)) throw new TypeError("inputs must be an object");
  const dispatch: Record<string, (value: AnyRecord) => AnyRecord> = {
    "D40-F05-A01": rocCurveAuc,
    "D40-F05-A02": precisionRecallAuc,
    "D40-F05-A03": brierScore,
    "D40-F05-A04": binaryLogLoss,
    "D40-F05-A05": reliabilityEce,
    "D40-F05-A06": gainsLift,
    "D40-F05-A07": costSensitiveThreshold,
    "D40-F05-A08": scoreMigration,
    "D40-F05-A09": sliceValidation,
    "D40-F05-A10": rareEventBounds,
  };
  const implementation = dispatch[topicId];
  if (!implementation) throw new RangeError(`unsupported topic_id: ${topicId}`);
  return implementation(inputs);
}

Playground

The embedded lab now expands to its full document height, keeping the article as the only scroll surface.

Full-height labplaygroundOpen full screen