Library/Market Data Engineering/Cleaning and Validation

D01-F02-A04 / Complete engineering topic

Stale-Quote Detector: Tell a Quiet Market from a Broken Clock

A production-minded guide to Stale-Quote Detector.

D01 · MARKET DATA ENGINEERING
D01-F02-A04Canonical / Tested / Open
D01 / D01-F02
Key concepts

The governed definitions this build depends on. Read them first if a term is unfamiliar.

Practical promise. Build a detector that can say which freshness dimension failed—without calling every unchanged quote an outage.

Four independent quote-freshness clocks feeding a separate business policy

A price can sit still while messages and heartbeats continue normally. A quote can change yet arrive too late. A feed can stop delivering packets while the last displayed bid and ask still look plausible. Those states need different evidence and different operational responses.

This tutorial builds a causal detector for one instrument and feed. It separates four clocks:

  1. source-event age at the current decision;
  2. transport age of the latest quote;
  3. local duration since bid or ask last changed; and
  4. local duration since any quote or heartbeat arrived.

Only after those measurements exist does the detector apply an active-session business policy. That separation is the main lesson.

The five questions the output must answer

Suppose the screen still shows 100.00 / 100.02. “Is it stale?” hides five questions:

QuestionDiagnosticWhat it can establish
How old is the source event now?Source-event ageThe last quote event is old at decision time
Did that quote arrive late?Transport ageSource-to-receiver delay exceeded policy
How long has the pair been unchanged locally?Unchanged durationBid/ask price did not change for a duration
Is the channel still producing data or heartbeats?Heartbeat ageExpected feed activity is absent
Does active-session policy call the quote stale?Business stalenessSession and activity context make raw age actionable

A heartbeat can keep liveness healthy while the quote remains unchanged. Conversely, the last quote may be recent but the absence of subsequent heartbeats can make the channel unusable. Neither condition proves what caused it.

Start with clock ownership

Each accepted event carries a receiver timestamp observed_ts. Quotes also carry a source timestamp source_event_ts.

The source and receiver clocks are different domains. Any subtraction across them assumes verified synchronization. That is why clock_sync_ok: true is a required quote field, not an optional annotation. FINRA Rule 6820 provides a concrete regulatory example of synchronization, drift checks, and documentation, although its CAT tolerances are not the thresholds used here.

Local durations are safer: the unchanged and heartbeat timers use only receiver-clock timestamps. In production, record a monotonic clock alongside wall-clock UTC so a time correction cannot make a duration run backward. The reference package rejects decreasing UTC observations and never silently clamps a negative age to zero.

The contract before the formula

The stream has three event types:

  • quote updates bid/ask state and also proves feed activity;
  • heartbeat proves feed activity without pretending the quote changed; and
  • check evaluates elapsed time without pretending a packet arrived.

Every event includes a session_id, session_state, and activity_expected. A new session_id resets all state. INACTIVE and HALTED suppress the business classification, while raw measurements remain visible.

A quote is rejected before state mutation when:

  • either timestamp is missing, invalid, or lacks a UTC offset;
  • source/receiver synchronization is unverified;
  • receive time precedes source time;
  • source timestamps decrease within the session;
  • bid or ask is missing, nonfinite, or nonpositive; or
  • receiver timestamps decrease.

This detector does not decide whether a crossed quote is valid; that is a different invariant handled by the Crossed/Locked Market Detector.

Four clocks, one policy

At event kk, define:

SymbolMeaningClock domain
ckc_kcurrent local evaluation timereceiver
eqe_qlatest quote's source-event timesource
rqr_qlatest quote's receive timereceiver
rΔr_\Deltareceive time of last bid/ask changereceiver
rmr_mreceive time of last quote or heartbeatreceiver

The four ages are:

sk=ckeq,q=rqeq,uk=ckrΔ,hk=ckrm.s_k=c_k-e_q,\qquad \ell_q=r_q-e_q,\qquad u_k=c_k-r_\Delta,\qquad h_k=c_k-r_m.

sks_k and q\ell_q cross the source and receiver clocks. uku_k and hkh_k stay inside the receiver domain.

Let SS be maximum source-event age, LL maximum transport age, UU the unchanged-duration threshold, and HH the heartbeat timeout. The reference equality rules are deliberately asymmetric:

sk>S,q>L,ukU,hk>H.s_k>S,\qquad \ell_q>L,\qquad u_k\ge U,\qquad h_k>H.

A maximum allowable age is still allowed at equality. A duration condition triggers when its boundary is reached. These choices are implementation policy, not universal market rules.

Business staleness is a separate gate:

business-stalek=activekexpectedk(sk>SukU).\text{business-stale}_k= \text{active}_k\land\text{expected}_k\land (s_k>S\lor u_k\ge U).

Transport lateness and heartbeat loss remain separate operational reasons. The result may contain more than one reason.

Work the boundary by hand

Use S=1200S=1200 ms, L=250L=250 ms, U=3000U=3000 ms, and H=1200H=1200 ms.

A quote with source time 09:30:00.000 arrives at 09:30:00.100. Its transport age is 100100 ms. At 09:30:01.200, its source-event age is exactly 12001200 ms, so the strict maximum-age rule has not fired.

A same-price quote later arrives at 09:30:02.500. It refreshes source-event age and heartbeat age, but it does not reset the unchanged anchor at 09:30:00.100. At 09:30:03.300, unchanged duration is 32003200 ms. The detector emits UNCHANGED_DURATION and, because the session is active and activity is expected, BUSINESS_STALE.

A changed pair received at 09:30:03.550 resets the unchanged duration. A later changed quote that takes 251251 ms to arrive emits TRANSPORT_AGE; price movement does not excuse late delivery.

Timeline showing equality, independent timer resets, and rejection of negative transport age

The diagram's lower branch is important: a source timestamp later than its receive timestamp is rejected as clock skew or timestamp error. It is never labeled “fresh” and never allowed to reset state.

Update state only after validation

Rendering system map…

The order prevents a malformed quote from refreshing the heartbeat or resetting the unchanged timer. Both implementations expose a stateful detector so tests can prove that state survives a rejected event unchanged.

Heartbeats are not synthetic quote changes

Nasdaq's SoupBinTCP 4.0 protocol distinguishes sequenced data from logical heartbeats. Its server heartbeat schedule is protocol evidence, not permission to use one second as a universal application timeout.

The reference detector treats either a quote or heartbeat as feed activity. Only a quote can change the bid/ask pair. A check advances neither anchor. That makes these two situations observable:

  • heartbeats continue while uku_k grows: the channel is alive, the value is unchanged;
  • neither data nor heartbeats arrive while activity is expected: hkh_k crosses its timeout.

Before any quote exists, quote-dependent ages are null, usability is false, and the reason is NO_QUOTE. Unknown is not fresh.

Calendar and session context change the meaning

At a regular-session close, a quote may remain unchanged for hours without constituting an active-market business failure. During a halt, the expected update pattern may differ again. The detector therefore accepts session context instead of embedding one exchange schedule.

source_event_age_exceeded may remain true outside the session; it is a raw measurement. business_stale becomes false when session_state is not ACTIVE or activity_expected is false. Heartbeat loss is also suppressed when activity is not expected.

The caller must provide a versioned exchange calendar, early closes, auctions, halts, and feed-specific service windows. This package cannot infer them from quote timestamps.

A real incident that shows why source identity matters

On August 22, 2013, UTP Vendor Alert 2013-9 records intermittent UQDF quote-channel outages at 11:48 ET and all outbound quote channels ceasing near 12:03. It reports that the technical issue was resolved and stable at 12:24, controlled connectivity return began at 14:00, and all Tape C quoting and trading resumed at 15:25. UTP trade feeds continued operating.

Nasdaq's separate trading post-mortem says proprietary market-data feeds were operating normally. The SEC's Regulation SCI release independently discusses the UTP SIP's inability to process quotes and the market-wide halt.

This is a documented quote-feed incident, not a generic flat-quote example. It teaches three precise lessons:

  • source identity matters because quote, trade, consolidated, and proprietary paths can differ;
  • technical repair time and the later coordinated market reopening are not the same interval; and
  • an unchanged quote alone could not have established this cause.

No Financial Modeling Prep data case is used. End-of-day OHLCV cannot validate packet heartbeat or transport behavior.

Implementations and parity tests

The Python implementation and TypeScript implementation share the same 12-event JSON fixture. They emit the same numeric ages, flags, and ordered reason codes.

Tests cover:

  • exact boundary values in both languages;
  • heartbeats that refresh liveness without resetting quote value;
  • active, inactive, halted, and new-session behavior;
  • negative age and unverified clocks rejected before mutation;
  • missing, naive, nonfinite, and out-of-order input; and
  • no-quote and empty-stream state.

Those tests establish conformance to this definition. They do not establish threshold optimality, predictive power, or profitability.

Explore the four failure modes

Open the interactive stale-quote lab. Its initial normal state is already rendered. Then compare:

  • Normal: quotes and heartbeats keep all clocks inside policy;
  • Unchanged: heartbeats continue while the bid/ask pair crosses UU;
  • Heartbeat loss: checks continue after all feed packets stop;
  • Clock skew: a negative transport age is rejected and prior state remains intact.

Use Back and Step to inspect one transition at a time. Reset always restores the same first accepted observation.

Failure modes that remain

  • A low-activity instrument can cross an unchanged threshold while entirely valid.
  • A heartbeat proves link activity, not economic quote freshness.
  • This variant ignores size-only changes.
  • Recent timestamps do not prove sequence completeness.
  • Wall-clock synchronization does not replace a monotonic duration clock.
  • One feed's condition says nothing about an independently sourced feed.
  • usable_for_active_market is an engineering diagnostic, not an order-routing recommendation.

Keep raw observations, source identifiers, clock-sync logs, calendar versions, and reason codes. A single stale=true field is too lossy for incident analysis.

Summary and next topic

A useful stale-quote detector does not ask one vague question. It measures four clocks, rejects invalid clock comparisons, preserves independent reasons, and applies business policy only with explicit session context.

Continue with Duplicate-Trade Resolver, where identity and lifecycle state replace elapsed-time thresholds but the same validation-before-mutation discipline applies.

Validation-before-mutation flow

This flow shows why clock errors are rejected before any freshness anchor changes.

Rendering system map…

Takeaway: rejection is a state-preserving outcome, not a stale/fresh classification.

ReferencesPrimary sources and evidence notes

Expand the source trail, evidence role, and limitations behind the engineering choices.

Accessed 2026-07-22. Detector thresholds and the final business policy are package-specific implementation choices, not venue rules.

NASDAQ-ITCH — Nasdaq TotalView-ITCH 5.0 Specification

  • Organization or authors: Nasdaq
  • Source type: Official exchange feed specification
  • Publication or effective date: Revision history latest entry June 13, 2022
  • Version: 5.0
  • URL or DOI: https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: Source messages use nanoseconds-since-midnight timestamps; system events and per-message source timestamps are distinct protocol fields.
  • Limitations: Does not define receiver timestamps, a universal stale-quote threshold, or this package's business policy.

NASDAQ-SOUP — SoupBinTCP 4.0

  • Organization or authors: Nasdaq
  • Source type: Official exchange transport protocol specification
  • Publication or effective date: Current document at access
  • Version: 4.0
  • URL or DOI: https://www.nasdaqtrader.com/content/technicalsupport/specifications/tradingproducts/soupbintcp4_0.pdf
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: Logical heartbeats are distinct from sequenced data; the server sends a heartbeat after more than one second without data so clients can detect extended silence.
  • Limitations: Protocol-specific behavior. It does not make one second a universal application timeout or prove quote-value freshness.

NASDAQ-2013 — Post Mortem: NASDAQ Trading Halted in Tape C Securities

  • Organization or authors: Nasdaq OMX
  • Source type: Official venue trader alert, Equity Trader Alert 2013-77
  • Publication or effective date: August 23, 2013
  • Version: Final post-mortem notice
  • URL or DOI: https://www.nasdaqtrader.com/TraderNews.aspx?id=ETA2013-77
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: UQDF quote dissemination ceased at 12:03 ET on August 22, 2013; the Tape C halt and restart timeline; Nasdaq proprietary feeds were reported as operating normally.
  • Limitations: Historical incident evidence, not a threshold calibration dataset. It cannot be generalized from a flat quote alone.

UTP-2013-9 — UTP SIP Issue on Thursday, August 22, 2013

  • Organization or authors: UTP SIP / Nasdaq OMX
  • Source type: Official UTP Vendor Alert 2013-9
  • Publication or effective date: August 23, 2013
  • Version: Final vendor alert
  • URL or DOI: https://www.nasdaqtrader.com/TraderNews.aspx?id=uva2013-9
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: Intermittent UQDF outages at 11:48 ET, all outbound quote channels ceasing near 12:03, technical stability at 12:24, controlled connectivity return at 14:00, and full Tape C quoting/trading resumption at 15:25; UTP trade feeds remained operational.
  • Limitations: Incident-specific timeline. The coordinated market reopening lasted longer than the technical repair and must not be labeled continuous technical downtime.

SEC-SCI — Regulation Systems Compliance and Integrity

  • Organization or authors: U.S. Securities and Exchange Commission
  • Source type: Final rule, Exchange Act Release No. 73639
  • Publication or effective date: November 19, 2014
  • Version: Final rule
  • URL or DOI: https://www.sec.gov/files/rules/final/2014/34-73639.pdf
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: Independent regulatory account of the August 22, 2013 Nasdaq SIP failure and the importance of resilient market systems.
  • Limitations: Regulatory systems context, not an implementation specification for this detector.

FINRA-6820 — Clock Synchronization

  • Organization or authors: Financial Industry Regulatory Authority
  • Source type: Current rule text
  • Publication or effective date: Adopted effective March 15, 2017; current at access
  • Version: FINRA Rule 6820
  • URL or DOI: https://www.finra.org/rules-guidance/rulebooks/finra-rules/6820
  • Accessed: 2026-07-22
  • Jurisdiction: United States
  • Supports: Business clocks require synchronization, monitoring, resynchronization, and documented drift controls.
  • Limitations: Applies to covered CAT business clocks. Its tolerances are not stale-quote defaults and are not imported into this package.
staleQuote.ts
export type SessionState = "ACTIVE" | "INACTIVE" | "HALTED";
export type EventKind = "quote" | "heartbeat" | "check";

export type DetectorEvent = {
  kind: EventKind;
  observed_ts: string;
  session_id: string;
  session_state: SessionState;
  activity_expected: boolean;
  source_event_ts?: string;
  clock_sync_ok?: boolean;
  bid?: number;
  ask?: number;
  [key: string]: unknown;
};

export type DetectorConfig = {
  max_source_event_age_ms: number;
  max_transport_age_ms: number;
  unchanged_threshold_ms: number;
  heartbeat_timeout_ms: number;
};

export const DEFAULT_CONFIG: DetectorConfig = {
  max_source_event_age_ms: 1200,
  max_transport_age_ms: 250,
  unchanged_threshold_ms: 3000,
  heartbeat_timeout_ms: 1200,
};

export class ClockDomainError extends Error {}

function milliseconds(value: unknown, field: string): number {
  if (typeof value !== "string" || !/(Z|[+-]\d\d:\d\d)$/.test(value)) {
    throw new Error(`${field} must be an ISO-8601 timestamp with a UTC offset`);
  }
  const parsed = Date.parse(value);
  if (!Number.isFinite(parsed)) throw new Error(`${field} must be a valid ISO-8601 timestamp`);
  return parsed;
}

function finitePrice(value: unknown, field: string): number {
  if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) {
    throw new Error(`${field} must be a finite positive number`);
  }
  return value;
}

function validateConfig(config: DetectorConfig): void {
  if (!Object.values(config).every((value) => Number.isFinite(value) && value >= 0)) {
    throw new Error("all thresholds must be finite and nonnegative");
  }
}

export class StaleQuoteDetector {
  private readonly config: DetectorConfig;
  private lastObservedMs: number | undefined;
  private sessionId: string | undefined;
  private lastMessageReceiveMs: number | undefined;
  private lastSourceEventMs: number | undefined;
  private lastTransportAgeMs: number | undefined;
  private lastPair: string | undefined;
  private lastChangeReceiveMs: number | undefined;

  constructor(config: DetectorConfig = DEFAULT_CONFIG) {
    validateConfig(config);
    this.config = {...config};
  }

  private resetSession(sessionId: string): void {
    this.sessionId = sessionId;
    this.lastMessageReceiveMs = undefined;
    this.lastSourceEventMs = undefined;
    this.lastTransportAgeMs = undefined;
    this.lastPair = undefined;
    this.lastChangeReceiveMs = undefined;
  }

  process(event: DetectorEvent) {
    if (!["quote", "heartbeat", "check"].includes(event.kind)) {
      throw new Error("kind must be quote, heartbeat, or check");
    }
    const observedMs = milliseconds(event.observed_ts, "observed_ts");
    if (typeof event.session_id !== "string" || !event.session_id.trim()) {
      throw new Error("session_id must be a nonempty string");
    }
    if (!["ACTIVE", "INACTIVE", "HALTED"].includes(event.session_state)) {
      throw new Error("session_state must be ACTIVE, INACTIVE, or HALTED");
    }
    if (typeof event.activity_expected !== "boolean") {
      throw new Error("activity_expected must be boolean");
    }
    if (this.lastObservedMs !== undefined && observedMs < this.lastObservedMs) {
      throw new Error("observed_ts must be nondecreasing");
    }

    let sourceEventMs: number | undefined;
    let transportAgeMs: number | undefined;
    let pair: string | undefined;
    if (event.kind === "quote") {
      if (event.clock_sync_ok !== true) {
        throw new ClockDomainError("quote rejected: source/receiver clock synchronization is not verified");
      }
      sourceEventMs = milliseconds(event.source_event_ts, "source_event_ts");
      transportAgeMs = observedMs - sourceEventMs;
      if (transportAgeMs < 0) {
        throw new ClockDomainError("quote rejected: negative transport age indicates clock skew or timestamp error");
      }
      pair = `${finitePrice(event.bid, "bid")}|${finitePrice(event.ask, "ask")}`;
      if (this.sessionId === event.session_id && this.lastSourceEventMs !== undefined && sourceEventMs < this.lastSourceEventMs) {
        throw new Error("source_event_ts must be nondecreasing within a session");
      }
    }

    if (this.sessionId !== event.session_id) this.resetSession(event.session_id);
    this.lastObservedMs = observedMs;
    if (event.kind === "quote" || event.kind === "heartbeat") this.lastMessageReceiveMs = observedMs;
    if (event.kind === "quote") {
      this.lastSourceEventMs = sourceEventMs;
      this.lastTransportAgeMs = transportAgeMs;
      if (pair !== this.lastPair) {
        this.lastPair = pair;
        this.lastChangeReceiveMs = observedMs;
      }
    }

    const sourceEventAgeMs = this.lastSourceEventMs === undefined ? null : observedMs - this.lastSourceEventMs;
    const heartbeatAgeMs = this.lastMessageReceiveMs === undefined ? null : observedMs - this.lastMessageReceiveMs;
    const unchangedDurationMs = this.lastChangeReceiveMs === undefined ? null : observedMs - this.lastChangeReceiveMs;
    const sourceEventAgeExceeded = sourceEventAgeMs !== null && sourceEventAgeMs > this.config.max_source_event_age_ms;
    const transportLate = this.lastTransportAgeMs !== undefined && this.lastTransportAgeMs > this.config.max_transport_age_ms;
    const unchangedThresholdReached = unchangedDurationMs !== null && unchangedDurationMs >= this.config.unchanged_threshold_ms;
    const heartbeatLost = event.activity_expected && heartbeatAgeMs !== null && heartbeatAgeMs > this.config.heartbeat_timeout_ms;
    const businessStale = event.session_state === "ACTIVE" && event.activity_expected && (sourceEventAgeExceeded || unchangedThresholdReached);
    const usable = event.session_state === "ACTIVE" && this.lastPair !== undefined && !businessStale && !transportLate && !heartbeatLost;
    const reasons: string[] = [];
    if (this.lastPair === undefined) reasons.push("NO_QUOTE");
    if (sourceEventAgeExceeded) reasons.push("SOURCE_EVENT_AGE");
    if (transportLate) reasons.push("TRANSPORT_AGE");
    if (unchangedThresholdReached) reasons.push("UNCHANGED_DURATION");
    if (heartbeatLost) reasons.push("HEARTBEAT_TIMEOUT");
    if (businessStale) reasons.push("BUSINESS_STALE");
    if (event.session_state !== "ACTIVE") reasons.push(`SESSION_${event.session_state}`);

    return {
      ...event,
      source_event_age_ms: sourceEventAgeMs,
      last_transport_age_ms: this.lastTransportAgeMs ?? null,
      unchanged_duration_ms: unchangedDurationMs,
      heartbeat_age_ms: heartbeatAgeMs,
      source_event_age_exceeded: sourceEventAgeExceeded,
      transport_late: transportLate,
      unchanged_threshold_reached: unchangedThresholdReached,
      heartbeat_lost: heartbeatLost,
      business_stale: businessStale,
      usable_for_active_market: usable,
      reasons,
    };
  }
}

export function detectStaleQuotes(events: DetectorEvent[], config: DetectorConfig = DEFAULT_CONFIG) {
  const detector = new StaleQuoteDetector(config);
  return events.map((event) => detector.process(event));
}
Full-height labplaygroundOpen full screen