Markov-Switching Autoregression

State models are attractive because they turn a noisy sequence into an interpretable evolving belief. They are dangerous for exactly the same reason: a smooth line or decisive regime label can look like fact even when it is only the output of a fragile configuration. This tutorial shows how to combine Markov regime probabilities with regime-specific AR(1) forecasts while keeping the information boundary and uncertainty visible.

The deliverable is practical. You will calculate one update, implement the recursion in Python and TypeScript, inspect eight model-specific scenarios, and use a guided lab to see why the state changes. The result is an educational filter, not an investment recommendation or a claim of predictive power.

The question this model answers

Markov-Switching Autoregression answers a conditional question: given the selected model, its fixed configuration, and observations available through the current timestamp, what state summary should be carried forward?

That wording matters. It does not ask what the state “really was” after seeing the future. It does not estimate parameters. It does not decide whether a trading action is profitable. It teaches how persistence and volatility can switch together while keeping filtered probabilities distinct from hindsight-smoothed labels.

Intuition before notation

Each regime proposes a different conditional forecast; transition odds and forecast errors decide how much posterior mass each proposal receives. Every update therefore needs a ledger. The learner should be able to point to the prior, the prediction, the new evidence, and the posterior. If one of those pieces is hidden, the final curve becomes an assertion rather than an explanation.

The state update visual is canonical: its labels correspond directly to the trace produced by both implementations.

The frozen definition

This article uses a two-regime fixed-parameter Hamilton AR(1) filter. Two-regime fixed-parameter AR(1) Hamilton filter with switching intercept, slope, and variance.

The core recursion is:

μⱼ,t=cⱼ+φⱼyₜ₋₁; αₜ(j)∝[Σᵢαₜ₋₁(i)pᵢⱼ]·N(yₜ|μⱼ,t,σ²ⱼ).

The package excludes maximum-likelihood fitting, smoothed probabilities, higher-order ar terms, time-varying transitions, and empirical recession calls. Those are legitimate extensions, but adding them silently would make the arithmetic and information set ambiguous.

The package convention that changes the answer

At the first row, where no lagged observation exists, each regime starts at c_j/(1-phi_j). Later rows use c_j+phi_j*y_(t-1). This intercept form is not the same parameterization as software that centers the AR process around a regime mean.

That sentence belongs beside the formula because two implementations can both be called Markov-Switching Autoregression and still disagree when they initialize, index, or parameterize the recursion differently.

When this model is the right abstraction

Use MSAR when regimes change the conditional dynamics of y_t given y_(t-1). A plain Gaussian HMM omits that lagged forecast channel.

Normalized probabilities can still be confidently wrong when the intercepts, AR slopes, variances, or transition matrix are stale. The practical selection test is not which curve looks smoothest. It is which latent representation, observation model, and information clock match the decision you must audit.

Filtering is not smoothing

A filtered state at t conditions on observations through t. A smoothed state at t can use later observations. Current statsmodels state-space documentation makes the same distinction between filtered and smoothed output (statsmodels 0.14.6 state-space guide).

For an online dashboard, alert, or causal backtest, silently substituting smoothed probabilities is a future-information leak. A retrospective research chart may use smoothing, but it must say so.

Inputs and clocks

The fixture uses 160 daily UTC observations per scenario. They are deterministic and synthetic. The timestamps are observation and availability times together; no exchange calendar, revision process, or vendor feed is implied.

The latent truth or true regime appears only for evaluation. The filter receives the observation values and fixed configuration. This separation lets us ask whether the recursion detects the generator without giving it the answer.

The configuration is part of the claim

{
  "transition": [
    [
      0.95,
      0.05
    ],
    [
      0.1,
      0.9
    ]
  ],
  "intercepts": [
    0.12,
    -0.18
  ],
  "phis": [
    0.72,
    -0.3
  ],
  "stds": [
    0.35,
    1.05
  ],
  "initial": [
    0.85,
    0.15
  ]
}

These values are implementation choices, not fitted estimates. A production claim would need a pre-declared fitting sample, likelihood or loss, optimizer, parameter constraints, model-selection process, and out-of-sample evaluation.

One update by hand

The first canonical observation is -0.07104479. Applying the published recursion gives regime 0 forecast = 0.42857, regime 1 forecast = -0.13846, posterior state 0 = 0.83413, posterior state 1 = 0.16587, next mixture forecast = 0.02539.

The calculation retains full precision and rounds only for display. The validator repeats this first update independently rather than asking the Python implementation to grade itself.

What the state ledger reveals

For this topic, the most useful fields are regime 0 forecast, regime 1 forecast, posterior state 0, posterior state 1, next mixture forecast. Read them together. A posterior without its predicted state hides persistence; a posterior without its innovation or likelihood hides the observation's contribution.

Eight scenarios that target this algorithm

The scenarios are not a generic family checklist. Each one changes a specific assumption that matters to Markov-Switching Autoregression:

• Switching persistence: Regimes differ in intercept, AR slope, and variance. The lab asks: Which regime forecast better explains each observation?
• Opposite AR slopes: Positive and negative serial dependence alternate. The lab asks: Can the filter distinguish regimes from forecast direction?
• Variance-only regimes: Conditional means match while innovation scales differ. The lab asks: Why should posterior certainty remain limited after one quiet row?
• Overlapping forecasts: Regime forecasts are deliberately close. The lab asks: Do probabilities communicate ambiguity better than a hard label?
• Fast switching: Truth changes regimes faster than the frozen transition matrix. The lab asks: How much lag comes from transition persistence?
• Rare regime: A brief alternative dynamic competes with strong prior odds. The lab asks: What forecast error is needed to identify a rare regime?
• Single outlier: One shock favors the high-variance regime without changing dynamics. The lab asks: Does a spike create only a temporary probability jump?
• Coefficient mismatch: Truth uses different intercepts and slopes from the filter. The lab asks: Can normalized probabilities still be confidently wrong?

This is the difference between a demo and a learning reference: the reader can name the stressed assumption before looking at the output.

The algorithm

1. Validate observations and the fixed configuration.
2. Initialize the declared prior.
3. Predict before reading the current observation.
4. score the observation under the predicted state or hypotheses.
5. update and normalize.
6. record diagnostics and carry the filtered state forward.

This order is causal. Reordering it can leak the observation into the prior or apply the transition twice.

A diagnostic that can disagree with the picture

Compare both regime-specific one-step forecasts with the observation, combine their likelihoods with the transition prior, then score the resulting probability rather than only its winning label.

The fixture-backed diagnostic is forecast probability quality and regime ambiguity. Brier score, negative log score, entropy, and a neutral 0.5 benchmark are paired with the two regime forecasts to expose coefficient and transition mismatch.

For the canonical full path, the published values include brier score state 1 0.0296, neutral probability brier 0.2500, negative log score 0.1291, mean posterior entropy nats 0.1960, most likely state accuracy 0.9812. They are author-derived from known synthetic truth after the causal trace is complete. They are not an empirical result and do not establish usefulness on market data.

The interactive version recalculates the same quantities only over rows revealed through the current step. That makes the score informative without letting future truth influence an earlier filter state.

Python and TypeScript parity

Both implementations accept the same observation list and configuration object. Both return one trace row per observation. The shared JSON fixture compares every primary state field at a tolerance of 1e-9.

Readable reference arithmetic is intentional. Optimization comes after the contract is proved. In production, use stable matrix decompositions, monitored conditioning, and explicit schema/version controls appropriate to the chosen extension.

The model-specific audit discipline

Continuous-state filters must keep covariance nonnegative. Discrete-state filters must keep probabilities normalized. The UKF must publish its sigma-point weights. BOCPD must define whether a change occurs before or after the current observation. MSAR must define transition direction and which regime owns each AR coefficient.

These are not cosmetic implementation details. They determine the answer.

For this topic, the decisive audit is: Compare both regime-specific one-step forecasts with the observation, combine their likelihoods with the transition prior, then score the resulting probability rather than only its winning label.

The outlier trap

A single extreme observation can produce a large innovation, strong emission ratio, or elevated change probability. That evidence is not the same as a persistent break. Compare the next several observations before attaching a narrative.

The failure map is a decision aid: check model assumptions, sensitivity, and subsequent evidence before treating a filtered state as operational truth.

Weak separation is an honest result

When regimes have similar emissions or observation noise dominates state movement, posterior probabilities should remain mixed. Forcing a hard label throws away the most important output: the model is uncertain.

The same principle applies to continuous filters. A narrow covariance is only meaningful if process and measurement variances are credible.

Historical evidence decision

The historical-example decision is not useful. A named market episode would add narrative appeal but would not validate a frozen latent-state model. The package therefore uses deterministic synthetic data so the hidden truth, observation clock, and state transition are auditable. A historical case would require licensed point-in-time observations, a parameter-estimation cutoff, revision policy, benchmark, and redistribution permission.

This avoids an especially common error: showing one named episode, fitting or tuning with knowledge of the whole episode, and then presenting a hindsight state chart as causal detection evidence.

Explore the guided lab

The self-contained playground implements choose → step → observe → explain → compare. Change the scenario and one meaningful parameter, then watch the trace, chart, guidance, and audit table recompute together.

Reset restores the canonical first row. Step and Play share one transition. In reduced-motion mode, Play advances exactly one observation and never starts a timer.

How to validate a production extension

• Freeze the information clock and revision policy.
• Separate fitting, filtering, forecasting, smoothing, and decision evaluation.
• Use time-ordered holdouts or rolling origins.
• Benchmark against a simpler state or regime rule.
• Test parameter sensitivity and transition/emission misspecification.
• Monitor numerical conditioning and probability normalization.
• Keep uncertainty visible in the user interface.
• Never convert a latent-state estimate directly into a performance claim.

Family map

The next tutorial is Bayesian Change-Point Detection. The representation changes, but the audit question stays constant: exactly what did the model know when it produced this state?

Summary

You can now implement and audit Markov-Switching Autoregression under a precise selected contract. The essential skill is not drawing a smooth state line. It is preserving the prior, prediction, evidence, posterior, and information boundary so another analyst can reproduce the result and challenge the assumptions.

Primary methodological records: A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle, MarkovAutoregression, Strictly Proper Scoring Rules, Prediction, and Estimation. The complete source-role and limitation ledger is in the canonical package.