Feed-Latency Monitor: Separate Delay from Clock Error

A latency number is meaningful only when you can name who created each timestamp, where it was captured, and whether the clocks were comparable. This tutorial builds that evidence into the result.

Imagine a market-data record stamped at the source at 13:30:00.950000 UTC and captured at your network interface at 13:30:00.970842 UTC. Subtraction gives 20.842 ms. That arithmetic is easy. The hard questions are operational:

• Did the source stamp transmission, matching, publication, or vendor receipt?
• Did your consumer stamp the NIC, kernel, parser, or strategy queue?
• Were those two clocks synchronized, and what was their error bound then?
• Does a negative result mean “faster than zero,” or does it invalidate the measurement?

A useful feed-latency monitor keeps those questions beside the number. It reports observed delay, clock quality, threshold uncertainty, and sequence continuity as separate evidence.

Start with ownership, not subtraction

This package uses three timestamps:

These names are a local data contract, not universal market terminology. For example, FIX Session Layer gives SendingTime(52) a precise owner: the session processor sets it when transmitting the message, not when queuing it, uses UTC, and expects comparison to a synchronized receiver clock (FIX Session Layer, section 4.2.3). A feed timestamp with another definition must retain that other definition.

The subtraction from event_time to receive_time does not isolate network transit. It includes every unobserved stage between the two capture boundaries. Calling it “exchange latency” or “network latency” would attribute cause the data cannot establish. The package calls it observed source-to-ingress latency.

UTC is a format; synchronization is evidence

Two timestamps can both end in Z and still come from clocks with different offsets. Let the physical path duration be $L_i^{physical}$. If the source and receiver clocks have offsets $o_s(t)$ and $o_r(t)$ from a common reference, then the measured difference is:

$$L_i^{obs}=L_i^{physical}+o_r(t_i)-o_s(t_i)$$

The offset difference looks exactly like latency. Clock skew is that offset at a moment. Clock drift is how the offset changes over time. A clock that passed a morning check can move outside its tolerance later.

That is why the input has a timestamped clock_profile, not a Boolean named synced. It names the source, ingress, and ready clock domains; the owners of the three timestamps; synchronization status; and the maximum absolute cross-domain error. FINRA Rule 4590 offers a useful operational example: its clock tolerance accounts for the reference difference, transmission delay, and drift, and it requires checks and logs throughout the day (FINRA Rule 4590). Its regulatory tolerances are context, not this lab’s alert thresholds.

NIST describes offset and uncertainty as explicit outputs of time comparison, including for financial-market and data-center users (NIST Time Measurement and Analysis Service). Precision without an uncertainty statement is not proof of accuracy.

Three durations, one uncertainty interval

For event $i$, define:

$$L_i^{obs}=t_i^{ingress}-t_i^{source}$$ $$P_i=t_i^{ready}-t_i^{ingress}$$ $$E_i^{obs}=L_i^{obs}+P_i$$

L_obs measures across source and consumer clock domains. P uses the two consumer timestamps and is valid as an ordinary duration only when those timestamps share a clock domain or have another declared comparison contract. E_obs is the sum of the two observed components, not a claim about a single owner.

If the declared maximum absolute cross-domain error is $\epsilon$, retain:

$$L_i^{physical}\in[L_i^{obs}-\epsilon,\ L_i^{obs}+\epsilon]$$

The interval does not “correct” the observation. It shows what the synchronization evidence permits. If a 5 ms warning threshold sits inside that interval, the correct state is threshold_uncertain, not an arbitrary ok or warning.

Negative latency is a clock-quality state

A monitor sometimes receives receive_time < event_time. Taking the absolute value, clamping to zero, or dropping the row hides the best evidence that the measurement contract failed.

This package emits clock_error_negative_latency, keeps the signed value and raw timestamps, and excludes it from the distribution. The state says only what the evidence proves: the observed cross-domain ordering is impossible as a physical one-way duration. Possible causes include clock offset, drift, an incorrectly named source timestamp, a mismatched capture layer, correction error, or reordered data. The monitor cannot choose among them without more evidence.

A reversed local pair, process_time < receive_time, receives a different state: processing_order_error. That distinction matters because it points to a consumer-side ordering or clock-domain problem rather than the cross-domain path.

Decision sequence

Notice the order: clock quality gates the distribution. A percentile built first and annotated later has already admitted observations that may not be comparable.

The sequence check is separate. A gap can coexist with an ok latency, and a clock error can occur with continuous sequences. Do not collapse completeness and timing into one traffic-light state.

Verify one event by hand

Synthetic event E0039 contains:

The observed source-to-ingress duration is 20.842 ms. Local processing adds 0.934 ms, for 21.776 ms observed end to end. The synthetic clock profile declares $\epsilon=0.05$ ms, so the bounded transport interval is 20.792–20.892 ms. Every value in that interval exceeds the pedagogical 15 ms critical threshold.

The 0.05 ms bound is part of the synthetic fixture. It is not a claim about a venue, vendor, network, or NIST service.

At the other edge, E0113 has an observed latency of -0.700 ms. The row remains visible as clock_error_negative_latency, and it never enters the p50, p95, p99, or maximum calculation.

Summarize only eligible observations

For sorted eligible values $x_{(0)},\ldots,x_{(n-1)}$, the implementation names its percentile convention. At probability $p$, use $h=(n-1)p$, $j=\lfloor h\rfloor$, then linearly interpolate:

$$Q_p=x_{(j)}+(h-j)(x_{(j+1)}-x_{(j)})$$

On the 180-event corpus, 174 events are ok, two are warning, three are critical, and one is clock_error_negative_latency. The eligible distribution is:

The stream also jumps from sequence 10075 to 10077. That is evidence of a discontinuity, not proof of permanent loss; retransmission and recovery rules are protocol-specific.

Use the lab to see the boundary move

Open the interactive feed-latency lab. It has three deterministic scenarios:

• Canonical burst stream replays the 180-event fixture and exposes the p99 tail.
• Threshold edge puts the clock-error interval across the warning boundary, producing threshold_uncertain.
• Clock failure shows negative cross-domain latency and a reversed consumer pair as explicit diagnostic states.

Choose a scenario, Step forward, Back up, or Play through the same transition function. Pause stops without changing state. Reset restores the scenario’s exact starting state. Changing thresholds recomputes all visible classifications rather than recoloring a cached result. Reduced-motion mode makes Play advance one complete step.

Implementation details that prevent hidden errors

The Python implementation uses timezone-aware datetime. The TypeScript implementation converts timestamps to integer epoch microseconds with bigint before subtraction. Subtracting large JavaScript epoch millisecond floats first can erase part of a microsecond-scale difference.

Both implementations:

1. validate thresholds and the clock profile;
2. preserve signed observed durations;
3. assign the ordered diagnostic state;
4. retain lower and upper transport bounds;
5. exclude clock and boundary-uncertain states from summaries;
6. detect sequence discontinuities; and
7. use the same linear percentile rule.

The tests assert exact fixture statuses and summaries, the E0039 arithmetic, negative latency, unknown clock synchronization, a threshold crossed by uncertainty, consumer timestamp reversal, malformed UTC, duplicate IDs, invalid thresholds, and empty input. Passing these tests establishes definition-level and cross-language parity. It does not prove a production deployment’s clocks are trustworthy.

A real incident that motivates—but cannot validate—the monitor

In 2012 the SEC reported that NYSE sent data through two proprietary feeds before sending it to consolidated feeds over an extended period beginning in 2008. The release described different internal paths and a software issue, with disparities ranging from single-digit milliseconds to multiple seconds (SEC release 2012-189).

That is an excellent explanation of why path timing matters. It is not an event-level worked example for this algorithm. The public release does not provide a redistributable set of per-message source, consolidated-path, vendor, and consumer timestamps together with clock ownership, capture layers, and synchronization bounds. Manufacturing those fields would turn a documented incident into fictional precision.

The evidence decision is therefore explicit: retain the case as historical motivation, and defer a factual numerical case until the raw timestamp chain and clock-quality evidence are public and reproducible.

Production checklist

• Name every timestamp owner and capture layer.
• Partition summaries by venue, feed, channel, session, and timestamp contract.
• Record synchronization status, bound, measurement time, holdover, offset, and drift evidence.
• Keep raw timestamps and signed observations immutable.
• Make clock_untrusted, threshold_uncertain, and negative latency visible.
• Keep latency, sequence, completeness, and repair policy separate.
• Calibrate warning and critical thresholds to an owned service objective; do not copy regulatory tolerances.
• Alert on the rate of excluded clock states as well as on p99 latency.
• Treat synthetic results as mechanics validation, never as provider performance evidence.

Summary

A trustworthy latency monitor is a clock-quality monitor first. It can tell you the observed time between named boundaries, the uncertainty allowed by synchronization evidence, the classification that follows, and whether the message sequence stayed continuous. It cannot identify an unobserved hop or convert an outage report into endpoint latency.

Continue with Price-Source Consensus Check to learn how timestamp eligibility and provenance affect agreement across multiple data sources.

Primary references

• FIX Session Layer, June 2020
• FINRA Rule 4590: Synchronization of Member Business Clocks
• NIST Time Measurement and Analysis Service
• SEC approval of the Consolidated Audit Trail plan
• SEC action on NYSE market-data distribution

Detailed source roles and limitations are recorded in REFERENCES.md.