D01-F04-A06 / Complete engineering topic

Provider Adjustment-Basis Drift Detector

A production-minded guide to Provider Adjustment-Basis Drift Detector.

D01 · MARKET DATA ENGINEERING
D01-F04-A06Canonical / Tested / Open
D01 / D01-F04

An adjusted close often enters a research pipeline as if it were an immutable fact. It is not. It is a provider-produced transformation whose value can change when a split or dividend is learned, corrected, reclassified, or processed under a revised methodology. If yesterday's backtest used one historical basis and today's rerun uses another, the result can move even when the research code did not.

This tutorial builds a control that separates an explainable restatement from unexplained basis drift. It compares two archived snapshots of the same provider field, uses only action evidence knowable between those snapshots, and emits a row-level audit trail.

Start with the identity, not the formula

A valid comparison pins provider, product, instrument, price field, and basis identifier. It also archives retrieval times and request parameters. Comparing a split-adjusted close from one endpoint with a split-and-dividend-adjusted close from another is not drift detection; it is a contract mismatch.

Alpha Vantage's official documentation distinguishes raw daily values from adjusted daily data containing split and dividend information. Its support material says its adjusted OHLCV convention uses both splits and cash dividends. Massive documents unadjusted flat files and separate adjusted REST choices. These examples show why “adjusted” must be qualified by provider and endpoint.

Fingerprint the basis

For date d in snapshot s, define:

F(d,s) = adjusted_price(d,s) / raw_price(d,s).

The observed multiplier between snapshots is:

M_obs(d) = F(d,new) / F(d,old).

A value of one means the factor did not change. A two-for-one split may produce a multiplier near 0.5 for older observations, depending on the provider convention. That change is not automatically an error.

Bring in knowledge time

An action can explain a revision only if it became available after the old snapshot and no later than the new one. The selected multiplier applies only to price dates before the action's effective date.

M_exp(d) = product of eligible action multipliers.

A future-discovered action cannot be used to excuse an earlier candidate snapshot. This small rule prevents a major look-ahead leak.

Measure the unexplained residual

The detector uses an absolute log ratio:

R_bps(d) = 10,000 × |ln(M_obs(d) / M_exp(d))|.

Multiplicative changes become symmetric and additive on the log scale. The declared tolerance is inclusive: a residual exactly equal to the threshold is accepted.

Adjustment-basis drift map

Work the synthetic example

Three baseline factors are 1.0. A newly knowable two-for-one split supplies expected multiplier 0.5. The new factors are 0.5, 0.5, and 0.5025.

The first two residuals are zero. The third is:

10,000 × |ln(0.5025 / 0.5)| = 49.875415 bps.

At a 1 bps tolerance, the first two rows are expected restatements and the third is drift. Crucially, the detector does not erase the expected rows or the action ID. An investigator can reproduce exactly which evidence explained each date.

Implementation shape

The reference implementation validates positive prices, unique dates, unique actions, timezone-aware knowledge times, and at least one shared date. It indexes snapshots, filters newly knowable actions, calculates factors and residuals, and aggregates row states. Python and TypeScript return the same JSON contract.

The complexity is linear after indexing and sorting dates. The more important engineering cost is archival: without immutable snapshots and request metadata, there is nothing point-in-time to compare.

Seven scenarios worth testing

The guided lab includes a canonical mixed audit, an exact expected split restatement, an unexplained one-percent change, an action learned too late, partial date overlap, a stable control, and a no-overlap failure. Each scenario contains 61 tolerance states. This makes a subtle truth visible: tolerance changes classification, but it cannot repair missing identity or missing overlap.

What this detector cannot prove

An alert does not prove that the provider is wrong. The action ledger may be incomplete, a symbol may have been remapped, currency may have changed, or a field may now follow a different documented basis. The correct response is investigation and dependency invalidation—not silently increasing tolerance.

Likewise, a zero residual proves only that the provider change matches the supplied multipliers. It does not prove total-return correctness, economic continuity, or predictive value.

Production checklist

Archive payload hashes, request parameters, endpoint version, provider entitlement, instrument identity, timezone, currency, and retrieval time. Version the corporate-action ledger and its source. Store the detector output with the research run it protects. Re-run dependent analytics only after an alert is classified and approved.

Evidence decision

The article uses synthetic values. A named historical provider case is deferred until both archived payloads, methodology versions, exact requests, action knowledge states, and publication rights are reproducible.

Next step

Corporate-Action Status and Effective-Date Reconciliation builds the governed event ledger this detector needs. Backward Split Adjustment then shows how to construct a selected split basis deliberately.

Primary sources and their precise roles are listed in REFERENCES.md.

Provider adjustment-basis drift calculation flow

Rendering system map…

The audit node states the invariant that must survive every derivative.

ReferencesPrimary sources and evidence notes

Expand the source trail, evidence role, and limitations behind the engineering choices.

S1 — Alpha Vantage API documentation

  • Organization: Alpha Vantage
  • Source type: Official provider documentation
  • URL: https://www.alphavantage.co/documentation/
  • Accessed: 2026-07-30
  • Supports: Raw daily series; adjusted daily values; split and dividend event fields.
  • Limitations: Provider-specific behavior; does not validate the synthetic fixture or drift detector.

S2 — Alpha Vantage support: adjustment method

  • Organization: Alpha Vantage
  • Source type: Official provider support documentation
  • URL: https://www.alphavantage.co/support/
  • Accessed: 2026-07-30
  • Supports: The provider states that adjusted OHLCV incorporates splits and cash dividends.
  • Limitations: Does not establish a universal adjustment convention.

S3 — Massive Stocks Splits API

  • Organization: Massive
  • Source type: Official provider API documentation
  • URL: https://massive.com/docs/rest/stocks/corporate-actions/splits
  • Accessed: 2026-07-30
  • Supports: Execution dates, split ratios, and historical adjustment factors.
  • Limitations: Provider-specific split treatment; no historical drift incident is claimed.

S4 — Massive Stocks flat-file overview

  • Organization: Massive
  • Source type: Official provider documentation
  • URL: https://massive.com/docs/flat-files/stocks/overview
  • Accessed: 2026-07-30
  • Supports: Flat files are unadjusted; adjusted REST data and split endpoints are separate choices.
  • Limitations: Current product documentation may change; pin the deployed version.

Publication decision

Documentation is publicly linkable. No provider payload is redistributed. All prices, dates, identifiers, action records, and results are synthetic or author-derived.

providerAdjustmentBasisDrift.ts
export interface PriceRow {
  date: string;
  raw_price: number;
  adjusted_price: number;
}

export interface AdjustmentAction {
  event_id: string;
  effective_date: string;
  available_at: string;
  status: string;
  adjustment_multiplier: number;
}

export interface AdjustmentDriftInput {
  provider: string;
  dataset: string;
  instrument_id: string;
  price_field: string;
  basis_id: string;
  baseline_observed_at: string;
  candidate_observed_at: string;
  tolerance_bps: number;
  baseline_rows: PriceRow[];
  candidate_rows: PriceRow[];
  actions: AdjustmentAction[];
}

export interface DriftRow {
  date: string;
  old_factor: number;
  new_factor: number;
  observed_multiplier: number;
  expected_multiplier: number;
  residual_bps: number;
  state: "stable" | "expected-restatement" | "basis-drift";
  action_ids: string[];
}

export interface AdjustmentDriftOutput {
  provider: string;
  dataset: string;
  instrument_id: string;
  price_field: string;
  basis_id: string;
  baseline_observed_at: string;
  candidate_observed_at: string;
  tolerance_bps: number;
  state: "stable" | "expected-restatement" | "basis-drift";
  overlap_count: number;
  stable_count: number;
  expected_restatement_count: number;
  drift_count: number;
  max_residual_bps: number;
  unmatched_baseline_dates: string[];
  unmatched_candidate_dates: string[];
  rows: DriftRow[];
}

const ACTIVE_ACTION_STATUSES = new Set(["confirmed", "completed", "corrected"]);

function timestamp(value: unknown, field: string): number {
  if (typeof value !== "string") {
    throw new TypeError(`${field} must be an ISO 8601 string`);
  }
  if (!/(Z|[+-]\d{2}:\d{2})$/.test(value)) {
    throw new Error(`${field} must include a timezone`);
  }
  const parsed = Date.parse(value);
  if (!Number.isFinite(parsed)) {
    throw new Error(`${field} must be a valid ISO 8601 timestamp`);
  }
  return parsed;
}

function day(value: unknown, field: string): string {
  if (
    typeof value !== "string" ||
    !/^\d{4}-\d{2}-\d{2}$/.test(value) ||
    Number.isNaN(Date.parse(`${value}T00:00:00Z`))
  ) {
    throw new Error(`${field} must be YYYY-MM-DD`);
  }
  return value;
}

function positive(value: unknown, field: string): number {
  if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) {
    throw new Error(`${field} must be finite and positive`);
  }
  return value;
}

function nonnegative(value: unknown, field: string): number {
  if (typeof value !== "number" || !Number.isFinite(value) || value < 0) {
    throw new Error(`${field} must be finite and nonnegative`);
  }
  return value;
}

function indexRows(value: unknown, field: string): Map<string, [number, number]> {
  if (!Array.isArray(value) || value.length === 0) {
    throw new Error(`${field} must be a non-empty list`);
  }
  const indexed = new Map<string, [number, number]>();
  value.forEach((rawRow, index) => {
    if (typeof rawRow !== "object" || rawRow === null || Array.isArray(rawRow)) {
      throw new TypeError(`${field}[${index}] must be an object`);
    }
    const row = rawRow as Record<string, unknown>;
    const date = day(row.date, `${field}[${index}].date`);
    if (indexed.has(date)) {
      throw new Error(`${field} contains duplicate date ${date}`);
    }
    indexed.set(date, [
      positive(row.raw_price, `${field}[${index}].raw_price`),
      positive(row.adjusted_price, `${field}[${index}].adjusted_price`),
    ]);
  });
  return indexed;
}

function round(value: number, digits: number): number {
  const scale = 10 ** digits;
  return Math.round((value + Number.EPSILON) * scale) / scale;
}

export function detectAdjustmentBasisDrift(
  input: AdjustmentDriftInput,
): AdjustmentDriftOutput {
  if (typeof input !== "object" || input === null || Array.isArray(input)) {
    throw new TypeError("input must be an object");
  }
  const identityFields = [
    "provider",
    "dataset",
    "instrument_id",
    "price_field",
    "basis_id",
  ] as const;
  for (const field of identityFields) {
    if (typeof input[field] !== "string" || input[field].length === 0) {
      throw new Error(`${field} must be a non-empty string`);
    }
  }

  const baselineTime = timestamp(input.baseline_observed_at, "baseline_observed_at");
  const candidateTime = timestamp(input.candidate_observed_at, "candidate_observed_at");
  if (candidateTime <= baselineTime) {
    throw new Error("candidate_observed_at must be after baseline_observed_at");
  }
  const toleranceBps = nonnegative(input.tolerance_bps, "tolerance_bps");
  const baseline = indexRows(input.baseline_rows, "baseline_rows");
  const candidate = indexRows(input.candidate_rows, "candidate_rows");
  if (!Array.isArray(input.actions)) {
    throw new TypeError("actions must be a list");
  }

  const seenActions = new Set<string>();
  const actions = input.actions.flatMap((action, index) => {
    if (typeof action !== "object" || action === null || Array.isArray(action)) {
      throw new TypeError(`actions[${index}] must be an object`);
    }
    if (typeof action.event_id !== "string" || action.event_id.length === 0) {
      throw new Error(`actions[${index}].event_id must be non-empty`);
    }
    if (seenActions.has(action.event_id)) {
      throw new Error(`duplicate action event_id ${action.event_id}`);
    }
    seenActions.add(action.event_id);
    const availableAt = timestamp(action.available_at, `actions[${index}].available_at`);
    const effectiveDate = day(action.effective_date, `actions[${index}].effective_date`);
    const multiplier = positive(
      action.adjustment_multiplier,
      `actions[${index}].adjustment_multiplier`,
    );
    if (
      ACTIVE_ACTION_STATUSES.has(action.status) &&
      baselineTime < availableAt &&
      availableAt <= candidateTime
    ) {
      return [
        {
          event_id: action.event_id,
          effective_date: effectiveDate,
          adjustment_multiplier: multiplier,
        },
      ];
    }
    return [];
  });

  const overlap = [...baseline.keys()]
    .filter((date) => candidate.has(date))
    .sort();
  if (overlap.length === 0) {
    throw new Error("snapshots must share at least one observation date");
  }

  const counts = { stable: 0, "expected-restatement": 0, "basis-drift": 0 };
  let maxResidual = 0;
  const rows: DriftRow[] = overlap.map((date) => {
    const [oldRaw, oldAdjusted] = baseline.get(date)!;
    const [newRaw, newAdjusted] = candidate.get(date)!;
    const oldFactor = oldAdjusted / oldRaw;
    const newFactor = newAdjusted / newRaw;
    const observedMultiplier = newFactor / oldFactor;
    const applicable = actions.filter((action) => date < action.effective_date);
    const expectedMultiplier = applicable.reduce(
      (product, action) => product * action.adjustment_multiplier,
      1,
    );
    const residualBps =
      Math.abs(Math.log(observedMultiplier / expectedMultiplier)) * 10_000;
    let state: DriftRow["state"];
    if (residualBps <= toleranceBps) {
      state =
        Math.abs(Math.log(expectedMultiplier)) <= 1e-15
          ? "stable"
          : "expected-restatement";
    } else {
      state = "basis-drift";
    }
    counts[state] += 1;
    maxResidual = Math.max(maxResidual, residualBps);
    return {
      date,
      old_factor: round(oldFactor, 10),
      new_factor: round(newFactor, 10),
      observed_multiplier: round(observedMultiplier, 10),
      expected_multiplier: round(expectedMultiplier, 10),
      residual_bps: round(residualBps, 6),
      state,
      action_ids: applicable.map((action) => action.event_id),
    };
  });

  const state: AdjustmentDriftOutput["state"] =
    counts["basis-drift"] > 0
      ? "basis-drift"
      : counts["expected-restatement"] > 0
        ? "expected-restatement"
        : "stable";

  return {
    provider: input.provider,
    dataset: input.dataset,
    instrument_id: input.instrument_id,
    price_field: input.price_field,
    basis_id: input.basis_id,
    baseline_observed_at: input.baseline_observed_at,
    candidate_observed_at: input.candidate_observed_at,
    tolerance_bps: toleranceBps,
    state,
    overlap_count: overlap.length,
    stable_count: counts.stable,
    expected_restatement_count: counts["expected-restatement"],
    drift_count: counts["basis-drift"],
    max_residual_bps: round(maxResidual, 6),
    unmatched_baseline_dates: [...baseline.keys()]
      .filter((date) => !candidate.has(date))
      .sort(),
    unmatched_candidate_dates: [...candidate.keys()]
      .filter((date) => !baseline.has(date))
      .sort(),
    rows,
  };
}

Full-height labplaygroundOpen full screen