# Missing-Bar Gap Classifier

A blank interval is an observation, not a cause. It might be outside the
session, covered by a trading halt, empty because no trades occurred, missing
because transport failed, or absent because a bar builder discarded valid
trades. Sometimes the available evidence cannot distinguish those causes. This
classifier preserves that uncertainty instead of filling the gap or guessing.

## Learning contract

This topic is for market-data engineers and quantitative analysts who already
understand UTC timestamps, half-open time bars, and exchange calendars. After
completing it, a reader can:

- build an auditable evidence ladder for one candidate slot;
- distinguish an observed bar from a missing bar and a zero-activity interval;
- explain why unknown or conflicting higher-priority evidence blocks a weaker
  inference;
- reproduce eight terminal classes in Python and TypeScript; and
- evaluate a historical closure without pretending to have provider prices.

The package diagnoses absence. It does not create prices, forward-fill bars,
construct a production exchange calendar, or claim that any repair is safe.

## Why a confidence-preserving classifier matters

An invented bar changes returns, volatility, rolling indicators, and execution
simulations. An invented cause is also dangerous: the same failed feed that
omitted a bar cannot independently prove that no trades occurred. Production
systems need evidence from distinct operational layers and must retain the
source identifiers that made a diagnosis possible.

Official sources illustrate why those layers exist. NYSE publishes calendar
and session information separately from its trading-halt information
([hours and calendars](https://www.nyse.com/trade/hours-calendars),
[trading halts](https://www.nyse.com/trade/trading-halts)). Nasdaq's MoldUDP64
protocol assigns message sequence numbers and defines retransmission requests
after a receiver detects a sequence-number gap
([MoldUDP64 specification](https://nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf)).
Those are sourced facts. The ordering policy below is this package's explicit,
conservative engineering choice.

## Terms and scope

| Term | Meaning |
|---|---|
| Candidate slot | A half-open interval `[t, t + delta)` in a versioned schedule or observed-bar union. |
| Evidence layer | One operational question: bar, session, market state, transport, or activity. |
| Positive fault evidence | A directly observed lost heartbeat or sequence gap; absence of a fault flag is not equivalent. |
| Independent activity | Trade evidence that does not depend on the same path whose bar is missing. |
| Unknown | Required evidence is absent, late, or not observable. |
| Conflicting | Two retained observations disagree at the same evidence layer. |

Classify per venue, instrument, bar size, calendar version, feed session, and
knowledge time. Consolidating venues before diagnosis can hide a local outage.

## Input contract

The shared CC0 synthetic fixture contains 15 decision states: six canonical
outcomes, five unknown paths, and four conflict paths. A state-machine lesson
needs 8–20 meaningful transitions; these 15 cases cover every branch without
padding the lab with decorative normal records.

| Field | Allowed values | Meaning |
|---|---|---|
| `bar_state` | `present`, `absent`, `unknown` | Whether a validated bar is materialized. |
| `session_status` | `open`, `closed`, `unknown`, `conflicting` | Result of versioned session evidence. |
| `halt_status` | `active`, `inactive`, `unknown`, `conflicting` | Official or authoritative market-state evidence. |
| `heartbeat_status` | `healthy`, `lost`, `unknown`, `conflicting` | Connection or feed-session health. |
| `sequence_status` | `continuous`, `gap`, `unknown`, `conflicting` | Sequenced-message continuity. |
| `activity_status` | `trades`, `no_trades`, `unknown`, `conflicting` | Whether valid trades occurred in the slot. |
| `activity_independent` | boolean | Whether activity came from a path independent of the missing bar. |
| `evidence_ids` | non-empty string array | Immutable identifiers for the observations used. |

All enum values are required. Unknown is data, not a null convention. A
production input also needs the instrument and venue identifiers, event-time
interval, receive and availability times, source and feed-session identifiers,
calendar version, correction version, and classifier version.

## Output contract

`diagnose_gap` returns an audit record rather than only a label:

| Field | Meaning |
|---|---|
| `classification` | One of the eight classes below. |
| `decisive_layer` | The strongest layer that terminated evaluation. |
| `reason` | Plain-language explanation of that decision. |
| `evidence_ids` | A copy of the input lineage identifiers. |
| `warnings` | Non-fatal cautions, such as dependent activity evidence. |

| Class | Interpretation | Repair implication |
|---|---|---|
| `present` | A validated bar exists. | Do not classify it as a gap, even if volume is zero. |
| `closed_session` | Authoritative session evidence excludes the slot. | Remove it from the expected grid; do not synthesize a bar. |
| `expected_halt` | A halt interval covers an otherwise open slot. | Preserve the absence and halt lineage. |
| `feed_outage` | Positive transport-failure evidence exists. | Recover or reconcile source messages before rebuilding. |
| `inactive` | Healthy transport plus independent evidence says no trades occurred. | Keep the interval empty unless a documented downstream convention requires an empty bar. |
| `bar_builder_error` | Independent trades exist but the expected bar does not. | Rebuild from validated trades and retain the incident. |
| `unknown` | A stronger layer cannot be established. | Escalate or wait for versioned evidence; do not guess. |
| `conflicting_evidence` | Sources disagree within a layer. | Quarantine and reconcile the source versions. |

The convenience function `classify_gap` returns only the class. It does not
replace the full diagnosis in an audited pipeline.

## Evidence hierarchy and decision rule

For slot `i`, let `B`, `S`, `H`, `T`, and `A` denote the consolidated states of
bar presence, session membership, halt state, transport health, and independent
activity. The classifier evaluates the following ladder:

1. **Bar observation:** `present` terminates as `present`; unknown bar presence
   terminates as `unknown`.
2. **Session evidence:** conflict terminates as `conflicting_evidence`; unknown
   terminates as `unknown`; closed terminates as `closed_session`.
3. **Market-status evidence:** conflict or unknown remains explicit; an active
   halt terminates as `expected_halt`.
4. **Transport evidence:** conflict remains explicit. A lost heartbeat or
   sequence gap is positive evidence for `feed_outage`. If neither is positive
   but either is unknown, the result is `unknown`.
5. **Activity evidence:** conflict or unknown remains explicit. Non-independent
   activity cannot prove inactivity, so it yields `unknown`.
6. **Construction:** independent `no_trades` yields `inactive`; independent
   `trades` with no bar yields `bar_builder_error`.

```mermaid
flowchart LR
    A["Bar evidence"] --> B["Session evidence"]
    B --> C["Market-status evidence"]
    C --> D["Transport evidence"]
    D --> E["Independent activity evidence"]
    E --> F["Construction diagnosis"]
    U["Unknown"] -. "stop; do not descend" .-> A
    U -.-> B
    U -.-> C
    U -.-> D
    X["Conflict"] -. "quarantine and reconcile" .-> B
    X -.-> C
    X -.-> D
    X -.-> E
```

The diagram shows evidence strength, not chronology. A later correction should
create a new diagnosis version rather than silently overwrite what was known at
an earlier cutoff.

### Pseudocode

```text
validate every enum, independence flag, and evidence identifier
if bar is present: return present
if bar is unknown: return unknown
for session then halt:
    if layer conflicts: return conflicting_evidence
    if layer is unknown: return unknown
    if terminal fact applies: return closed_session or expected_halt
if transport conflicts: return conflicting_evidence
if heartbeat lost or sequence gap: return feed_outage
if heartbeat or sequence unknown: return unknown
if activity conflicts: return conflicting_evidence
if activity unknown or depends on the missing path: return unknown
if independent activity is no_trades: return inactive
return bar_builder_error
```

## Worked state transitions

At 13:50 UTC in the fixture, the bar is absent, the calendar is open, no halt
is active, heartbeat is healthy, sequence continuity is intact, and an
independent activity source reports no trades. The ladder reaches activity and
returns `inactive`.

At 14:20 UTC, every upstream state is the same, but the zero-trade observation
comes from the missing path itself. `activity_independent` is false, so the
classifier returns `unknown`. The absence of messages on a failed path cannot
prove market inactivity.

At 14:25 UTC, two calendar versions disagree. The result is
`conflicting_evidence` at the session layer; later heartbeat and activity fields
cannot break that tie.

## Historical teaching case: a weekday that was officially closed

This case is intentionally narrow and separates four epistemic layers:

| Layer | Record |
|---|---|
| Sourced fact | ICE/NYSE announced on 1 December 2018 that named NYSE Group markets would be closed on Wednesday, 5 December 2018 for the National Day of Mourning ([official notice](https://ir.theice.com/press/news-details/2018/New-York-Stock-Exchange-to-Honor-President-George-H-W-Bush/default.aspx)). |
| Provider observation | None. The package does not claim to have inspected or licensed a vendor's bars. |
| Engineering assumption | Compare an intentionally naive weekday template of 09:30–16:00 America/New_York with five-minute half-open slots. |
| Derivation | The template creates `390 / 5 = 78` candidate slots in `[14:30Z, 21:00Z)`; the official closure creates zero expected slots. |
| Interpretation | A weekday-only grid would manufacture 78 apparent gaps. The correct diagnosis is `closed_session`, not outage or inactivity. |

The reproducible metadata and arithmetic are stored in
[`datasets/historical-closure-case.json`](datasets/historical-closure-case.json).
This case teaches calendar evidence; it does not validate provider coverage,
security-level halt data, or prices.

## Implementation and verification

Python and TypeScript implement the same pure, linear-time ladder. The shared
fixture checks every class and specifically proves that:

- positive transport faults can decide even when activity is unknown;
- an unknown stronger layer blocks a weaker no-trade inference;
- dependent zero-activity evidence stays unknown;
- evidence lineage and decisive layer survive the diagnosis; and
- invalid enums, false booleans, missing fields, and empty lineage fail closed.

The interactive lab exposes each branch through **Canonical**, **Unknowns**,
and **Conflicts** scenarios. Step, Back, Play, Pause, and Reset use the same
deterministic transition function. The static evidence ladder remains available
when scripts or motion are disabled.

## Failure modes and limitations

- A heartbeat proves only the monitored connection state; it does not prove
  that every payload arrived. That is why sequence continuity is separate.
- A sequence gap proves missing transport messages, not which economic events
  they contained. Recovery must precede bar reconstruction.
- A halt feed may arrive late or be revised. Retain availability time and emit a
  new diagnosis version.
- `closed_session` depends on the venue, product, session scope, and calendar
  version. Core and extended hours are not interchangeable.
- `inactive` is intentionally difficult to prove. If independence is uncertain,
  return `unknown`.
- The synthetic fixture proves definition and cross-language parity, not the
  frequency of these causes in live markets.
- The historical closure case has no provider-bar observation and must not be
  cited as evidence that a particular vendor behaved correctly.
- The labels are operational diagnostics, not trading signals and not a repair
  authorization.

## Related catalog topics

- Time Bars defines the candidate intervals.
- Exchange-Calendar Alignment supplies versioned session membership.
- Feed-Latency Monitor distinguishes late data from absent data.
- Point-in-Time Availability Guard preserves when revised evidence became
  knowable.

Source roles, dates, applicability, and limitations are recorded in
[`REFERENCES.md`](REFERENCES.md).