Missing-Bar Gap Classifier: Diagnose Absence Without Inventing Certainty

Practical promise. Given an empty time-bar slot, determine what the evidence actually supports: closed session, expected halt, feed outage, inactivity, bar-builder error, unknown, or conflict. You will also learn when the honest answer is “we cannot know yet.”

A blank candle is visually simple and operationally ambiguous. It might be absent because the market was closed. Trading might have been halted. The market might have been open but inactive. Messages might have been lost. Or the trades might be present while the bar builder failed.

Those causes lead to different actions. Removing an officially closed interval from an expected grid is sensible. Reconstructing a bar after a confirmed feed outage requires message recovery. Rebuilding a missing bar from independently captured trades addresses a construction defect. Forward-filling every blank interval does none of those things; it only makes uncertainty look numeric.

This article builds a conservative classifier that keeps observation, evidence, diagnosis, and repair separate. It is an operational data-quality method, not a trading signal.

One blank slot, five evidence layers

The first design decision is to stop calling all evidence “market data.” The classifier asks five different questions in order:

1. Is a validated bar actually present?
2. Does the versioned session calendar include the slot?
3. Does authoritative market-status evidence show a halt?
4. Was the source transport healthy and sequentially complete?
5. Did a source independent of the missing bar path observe trades?

NYSE, for example, publishes session/calendar information and halt information as distinct products and pages (NYSE hours and calendars, NYSE trading halts). At the transport layer, Nasdaq's MoldUDP64 specification defines sequence numbers and retransmission requests after a receiver detects a sequence-number gap (official specification).

Those sources establish that the evidence layers are real. They do not mandate this package's ordering. The precedence ladder is a conservative engineering policy and is tested as such.

Why unknown and conflicting are real outputs

A common classifier stores booleans such as halt=false or sequence_gap=false. That representation cannot distinguish “checked and false” from “not checked,” “late,” or “two sources disagree.” The result looks decisive even when the system never observed the decisive fact.

This package uses explicit states:

• a layer can be known and affirmative;
• known and negative;
• unknown; or
• conflicting.

Unknown means the evidence needed at that layer is absent, late, or not observable. Conflicting means retained sources disagree at the same layer. The classifier never treats either state as false.

This distinction matters most for inactivity. If a bar is absent from feed A, then “feed A reported zero trades” is not independent proof of no activity. Feed A may simply be the path that failed. The inactive class requires healthy transport and activity evidence independent of the missing bar path.

The input and output contracts

Each candidate slot supplies six categorical states, one independence flag, and immutable evidence identifiers. The exact schema is in the canonical README. In production, the record should additionally carry instrument, venue, event-time interval, receive and availability times, feed session, calendar version, correction version, and classifier version.

The primary output is an audit record:

The eight possible classes are present, closed_session, expected_halt, feed_outage, inactive, bar_builder_error, unknown, and conflicting_evidence.

Notice what is not in the output: a synthetic price. Classification should happen before any repair policy.

The conservative decision ladder

Let $B_i$, $S_i$, $H_i$, $T_i$, and $A_i$ denote the consolidated bar, session, halt, transport, and activity states for slot $i$. The classifier is an ordered function:

$$C_i = f(B_i, S_i, H_i, T_i, A_i).$$

The notation is deliberately compact because the enum contracts carry the meaning. Evaluation proceeds as follows:

1. A present bar returns present. Unknown presence returns unknown.
2. A session conflict returns conflicting_evidence; unknown returns unknown; closed returns closed_session.
3. A halt conflict or unknown remains explicit; an active halt returns expected_halt.
4. A transport conflict remains explicit. A lost heartbeat or sequence gap is positive evidence for feed_outage. If neither is positive but continuity is incomplete, the result is unknown.
5. Activity conflict or unknown remains explicit. Dependent activity evidence returns unknown with a warning.
6. Independent no-trade evidence returns inactive. Independent trades with no bar return bar_builder_error.

Takeaway: unknown at a stronger layer stops descent to a weaker story. A conflict is quarantined, not resolved by whichever source happens to arrive last.

Three cases that look similar on a chart

The static timeline compares evidence, not candle geometry:

Consider three missing in-session bars from the synthetic fixture.

Case 1: independent no-trade evidence

At 13:50 UTC, the session is open, no halt is active, heartbeat is healthy, sequence continuity is intact, and an independent activity source reports no trades. The classifier reaches the fifth layer and returns inactive.

Case 2: the same zero comes from the missing path

At 14:20 UTC, the visible states look similar, but the activity evidence is not independent. The classifier returns unknown and warns against using a failed path to prove inactivity.

Case 3: two calendars disagree

At 14:25 UTC, one retained calendar version says open and another says closed. The result is conflicting_evidence at the session layer. Healthy transport or zero activity cannot settle a disagreement about whether the slot belonged in the expected universe.

A historical example: 78 gaps that should never exist

Historical examples add value only when fact, provider observation, derivation, and interpretation stay separate. This one needs no vendor price record.

Sourced fact

On 1 December 2018, ICE/NYSE announced that the named NYSE Group markets would be closed on Wednesday, 5 December 2018 for the National Day of Mourning for President George H. W. Bush (official ICE/NYSE notice).

Provider observation

None. This package did not query or license a provider's historical bars. It therefore makes no claim about whether a particular vendor emitted, omitted, or later corrected records.

Engineering assumption and derivation

To expose the classifier's value, compare the official closure with an intentionally naive weekday template: 09:30–16:00 America/New_York in five-minute half-open slots. On that December date the stored example expresses the interval as [14:30Z, 21:00Z). The arithmetic is:

$$\frac{6.5\ \text{hours} \times 60\ \text{minutes/hour}} {5\ \text{minutes/slot}} = 78\ \text{slots}.$$

The naive template generates 78 candidate slots. The official closure produces zero expected slots.

Interpretation

A weekday-only grid would manufacture 78 apparent gaps for every instrument in scope. Those absences are not evidence of 78 feed failures. They are one calendar-model defect. With the closure notice attached, each naive candidate terminates at the session layer as closed_session.

The complete case metadata lives in historical-closure-case.json. The example demonstrates calendar diagnosis, not provider accuracy, price behavior, or economic impact.

Implementation: pure logic with auditable output

The Python and TypeScript implementations follow the same pattern:

1. validate every enum, the literal boolean independence flag, and non-empty evidence identifiers;
2. evaluate the ladder without mutating input;
3. copy evidence lineage into the diagnosis; and
4. return immediately at the first decisive layer.

The work is $O(n)$ time for $n$ candidate slots and $O(n)$ output space when the audit records are retained. The per-slot logic is constant space.

Both languages expose diagnoseGap or diagnose_gap for the audit record and a convenience classifyGap or classify_gap for the label. An audited production pipeline should keep the full result.

Use the lab to challenge the evidence

Open the interactive missing-bar evidence lab. It offers three scenarios:

• Canonical reaches the six confident outcomes;
• Unknowns shows where missing or dependent evidence stops evaluation; and
• Conflicts shows same-layer disagreements that require reconciliation.

Use Step and Back to move one decision at a time. Play and Pause call the same transition used by Step. Reset restores the same first case. The active ladder layer, exact field values, reason, evidence identifiers, and accumulated audit log remain synchronized. Reduced-motion mode converts Play to one deterministic step.

Tests that matter

The shared fixture contains 15 meaningful decision states rather than a long run of visually identical normal bars. Python and TypeScript tests verify:

• all eight terminal classes;
• full cross-language fixture parity;
• evidence lineage and decisive-layer preservation;
• a positive feed fault when activity is unknown;
• stronger unknown evidence blocking a weaker no-trade claim;
• dependent zero activity remaining unknown;
• invalid enum, type, missing-field, and lineage failures; and
• the exact 78-slot historical derivation with provider_observation: null.

Passing those tests proves that the implementation matches this package's definition. It does not prove how often each cause occurs in live markets or authorize a repair.

Production failure modes

• Boolean unknowns: storing absent halt metadata as false creates false confidence.
• Calendar drift: recomputing old slots with today's calendar silently changes history.
• Dependent inactivity evidence: zero messages on the failed path are not proof of zero trades.
• Heartbeat overclaim: a live connection does not prove that every payload arrived; sequence continuity remains separate.
• Halt timing: late or corrected market-status records require a new point-in-time diagnosis version.
• Venue consolidation: a consolidated bar can hide a single-venue failure.
• Repair inside diagnosis: filling a bar before preserving the cause destroys auditability.

Keep a repair decision in a separate field and workflow. closed_session might remove a slot from the expected grid. feed_outage might trigger replay and reconciliation. bar_builder_error might trigger deterministic reconstruction. unknown and conflicting_evidence should block automatic repair unless an explicit risk policy says otherwise.

Summary and next topic

A missing bar is a question. The evidence ladder answers only as strongly as the retained session, market-status, transport, and independent activity records allow. It can return a useful cause, but it can also preserve unknown or conflicting evidence without cosmetic certainty.

Continue with Feed-Latency Monitor to distinguish data that is absent from data that has merely not arrived yet.

The primary source roles, dates, and limitations are recorded in REFERENCES.md. The canonical README owns the contracts, and the implementations and shared fixture make them executable.