Measurement Error, Resolution, Accuracy, and Precision

A plain-language map from input to a checked conclusion

How close, repeatable, and finely recorded are the measurements? This sounds basic, but it is one of the most important questions in data work. A polished chart cannot rescue a dataset whose rows, fields, or time meaning were misunderstood.

The idea in one minute

Error is measured minus reference, accuracy concerns closeness to an accepted reference, precision concerns repeatability, and resolution is the smallest display or recording increment.

The safe rule is simple: Report bias and absolute error against a defensible reference, then report spread and resolution separately. This gives you an explanation that another person can inspect instead of a hidden assumption.

A tiny synthetic example

The package uses four deliberately small records so every conclusion can be checked by eye. Measurements 9.9, 10.0, 10.1, and 10.0 around reference 10 have zero average bias and mean absolute error 0.05. Zero bias does not mean every measurement is exact.

The independent result is bias = 0.00; mean absolute error = 0.05. “Synthetic” matters: these values teach the concept; they are not observations from a company, exchange, survey, or market-data provider.

The four-stage reasoning path used in this lesson

Use the four-stage check

Inspect. Read the fields and ask what one record represents.
Declare. Write the schema, grain, time basis, or measurement meaning that the calculation depends on.
Test. Run a small diagnostic that could expose a contradiction.
Explain. State both the result and its boundary.

This is intentionally more careful than “load a file and calculate.” It prevents the most dangerous data errors: the ones that return reasonable-looking numbers.

The tempting mistake

Using accuracy and precision as synonyms hides whether a process is centered, repeatable, both, or neither. The problem is semantic, so more decimal places or faster code will not fix it.

There is also an edge case: Rounded observations can appear perfectly repeatable while hiding variation below the recording resolution. A good pipeline exposes this state to the reader. It does not quietly select a convenient interpretation.

Try the guided lab

Open the self-contained guided lab. Choose Canonical, Edge, or Failure, then use Step to move from input through declaration, diagnostic, and explanation. The lab starts with useful data, works without a server, supports keyboard controls, and has a deterministic reduced-motion mode.

What this result does not prove

The diagnostic does not prove that the source is representative, error-free, licensed for every use, or fit for an investment decision. It tells you whether the narrow assumption in this lesson survives one explicit check. Unknown metadata is a reason to abstain, not permission to guess.

Optional code verification

Python and TypeScript implementations are included for reproducibility and use the same JSON expectation. They are optional: a nontechnical learner should be able to reach the same conclusion from the table and explanation alone.

Takeaway

Report bias and absolute error against a defensible reference, then report spread and resolution separately. If you can say what the input means, show the check, and name the boundary, your result is ready for the next analytical step.

Enhancement studio: draw, compare, explain

This additive studio does not replace the beginner lesson above. It gives you two more drawings, a decision comparison, and short practice prompts so you can explain the idea without copying a formula or writing code.

Drawing 1 — name, apply, check

Three-part concept anatomy for Measurement Error, Resolution, Accuracy, and Precision

Read left to right: name what the data means, apply the narrow lesson rule, then use an independent check. Open the full-size concept anatomy.

Choose the right idea

Decision	This lesson	Closest next or comparison	Why the difference matters
Main question	Error is measured minus reference, accuracy concerns closeness to an accepted reference, precision concerns repeatability, and resolution is the smallest display or recording increment.	Missing, Nonfinite, Censored, and Truncated Values	Choose the question before choosing the arithmetic.
Safe rule	Report bias and absolute error against a defensible reference, then report spread and resolution separately.	Uses its own input and boundary contract.	Neighboring lessons can use the same numbers but answer different questions.
Required check	bias = 0.00; mean absolute error = 0.05	Re-check its own unit, time, denominator, or schema.	A correct answer to the wrong question is still wrong.
Stop condition	Using accuracy and precision as synonyms hides whether a process is centered, repeatable, both, or neither.	Move only when its prerequisites are satisfied.	Unknown meaning is a reason to pause, not to guess.

Drawing 2 — common-mistake clinic

Safe reading compared with a tempting mistake for Measurement Error, Resolution, Accuracy, and Precision

The left side states the safe interpretation; the right side shows the mistake that often produces a believable but misleading result. Open the full-size mistake comparison.

Explain it back without code

Name it: What does the first input or observation mean?
Answer: Error is measured minus reference, accuracy concerns closeness to an accepted reference, precision concerns repeatability, and resolution is the smallest display or recording increment.
Choose it: Which rule belongs to this question?
Answer: Report bias and absolute error against a defensible reference, then report spread and resolution separately.
Challenge it: What check could make you stop?
Answer: bias = 0.00; mean absolute error = 0.05

If your explanation leaves out the unit, period, denominator, grain, or availability time that the lesson needs, it is not complete yet.

Related concepts and learning handoff

Governed glossary: measurement error, accuracy, precision, bias. Browse the full financial glossary when a term is unfamiliar.
Continue with: Missing, Nonfinite, Censored, and Truncated Values.
Evidence boundary: all displayed numbers remain synthetic teaching data; the drawings do not claim a market observation, forecast, or investment result.

Workflows

Rendered from the canonical Mermaid sources linked by this article.

Concept flow — D00-F03-A08

Rendering system map…

ReferencesPrimary sources and evidence notes

Expand the source trail, evidence role, and limitations behind the engineering choices.

The lesson uses primary standards, official statistical guidance, or official software documentation. The worked data are synthetic and author-derived.

1. NIST Glossary — Accuracy

URL: https://www.nist.gov/glossary-term/36846
Accessed: 2026-08-10
Supports: accuracy as closeness to an accepted reference and its relationship to trueness and precision.
Limitations: The accepted reference and uncertainty must still be justified for a specific dataset.
Source role: authoritative definition or implementation reference; no numerical teaching values were copied.

2. NIST Technical Note 1297 — Terminology

URL: https://www.nist.gov/pml/nist-technical-note-1297/nist-tn-1297-appendix-d1-terminology
Accessed: 2026-08-10
Supports: careful use of accuracy, precision, repeatability, reproducibility, and uncertainty terms.
Limitations: Terminology guidance does not replace a measurement-system study.
Source role: authoritative definition or implementation reference; no numerical teaching values were copied.

Evidence boundary

The sources support definitions and operational cautions. They do not validate a particular investment decision, provider dataset, or legal interpretation. The historical-example decision is not useful for this foundations lesson: a named market dataset would add licensing and point-in-time complications without making the core distinction clearer.

Implementation

Full dependency-light reference implementations in both supported languages.

algorithm.ts

import { runTopic as runD00Topic, type D00Input, type D00Output } from "../../../../shared/typescript/d00Engine.ts";

/** Run the canonical D00-F03-A08 calculation. */
export function measurementErrorResolutionAccuracyAndPrecision(input: D00Input): D00Output {
  return runD00Topic("D00-F03-A08", input);
}

Playground

The embedded lab now expands to its full document height, keeping the article as the only scroll surface.

Full-height labguided labOpen full screen