ADR-0022 — Declared coverage is a separate durable contract¶
Status: Accepted
Date: 2026-08-29
Context¶
The producer bridge from manifests to catalog.partitions is unimplemented, and
the semantics it would need are ambiguous in the current baseline.
docs/contracts/MARKET_DATA_INGEST_CONTRACTS.md §6 already separates observed
event bounds from declared coverage and leaves the population rule for
catalog.partitions.ts_start / ts_end explicitly open. DATA_GATEWAY.md §5
and §7 already consume those columns as declared half-open coverage, and
src/quant_platform/data/catalog.py prunes with them.
Three problems follow.
-
Nothing durable carries declared coverage.
partition-manifest-v1has onlyfirst_exchange_ts/last_exchange_ts, which are observed event bounds and are null for a zero-row partition.db/init/001_catalog.sqlopens with the principle that the database is rebuildable and that the truth lives in the manifests. Forts_start/ts_endthat principle is currently false: the values would exist only in PostgreSQL. -
The frozen schema's prose contradicts the newer contract.
partition-manifest-v1.jsoncalls the observed bounds "copertura", states that "la copertura temporale REALE resta descritta da first/last_exchange_ts", and says gaps and overlaps between partitions are detected from them. Read literally, that instructs a future producer to treat distance between events as a data gap. -
Several required cases have no representation at all: complete coverage with zero events, a collector interruption that no event pattern reveals, a sequence discontinuity as distinct from silence, and a gap for an interval where nothing was ever materialized.
Left unresolved, the bridge implementation would have to invent coverage semantics at runtime.
Decision¶
Freeze declared coverage as a separately versioned durable contract,
coverage-manifest-v1, keyed to the natural dataset identity. One document
records one acquisition or reconciliation operation: its intent, its versioned
source semantics and mapping, and one or more interval assertions each carrying
a status and source-specific evidence. Only status: complete declares
coverage, and it attributes to exactly one natural partition identity.
partition-manifest-v1 is not modified, in rules or in bytes. Its stale
wording is corrected by an errata table in
DECLARED_COVERAGE.md §7 rather than by
editing a frozen artifact.
The full semantics, the fifteen invariants and the reconstruction rule are in DECLARED_COVERAGE.md.
Alternatives considered¶
A — partition-manifest-v2 with declared-coverage fields. Rejected. It
fails three required cases structurally rather than inconveniently. A partition
manifest exists only once a partition is materialized, so a known gap over an
interval where nothing was written has nowhere to live. It has one interval slot
per partition, so an interrupted session inside one partition cannot be stated
truthfully. And because the document is keyed by (dataset, partition_key,
revision) and carries the content sha256, revising coverage after a repair
that changes no bytes would force a spurious revision N+1 and supersession.
It would also invalidate every existing manifest and fixture for a change the
sealed content did not require.
B — companion coverage sidecar per natural partition identity. Rejected. It solves the multiple-intervals-per-partition problem but keeps the other two: still nothing to attach to when no partition exists, and an operation spanning partitions must duplicate its intent and evidence per file. Amending a sidecar in place after a repair would also make the durable evidence mutable, with no supersession chain — the opposite of what provenance needs.
C — a separately versioned dataset-level contract. Chosen. Coverage is a
claim about an interval, so the artifact is keyed by interval and dataset, not
by file. It represents every required case, leaves partition-manifest-v1
untouched, keeps sealed content and coverage on independent lifecycles, and
gives supersession an explicit chain.
Within C, one document per operation was chosen over one document per
assertion: an interrupted live session must be able to state
complete / known_gap / complete as one truthful record, and intent, source
semantics and provenance are properties of the operation, not of each interval.
Catalog-only storage was excluded by the mandate and is independently wrong: it would contradict the rebuildability principle the catalog DDL already states.
Consequences¶
The future bridge is deterministic. It folds complete assertions per partition
and publishes only a single contiguous interval; it never inspects event
spacing, observed bounds or partition_key to obtain coverage.
Two failure modes become loud instead of silent. Copying observed bounds into
ts_start / ts_end is rejected by the containment invariant, because
half-open coverage always places last_exchange_ts == ts_end outside the
interval. And a partition whose attributed coverage is non-contiguous is
reported as unpublishable rather than published with its internal gap declared
covered.
That second case is a real constraint the producer inherits, and a
cross-boundary audit against the merged Consumer API baseline sharpened it.
Marking the partition degraded does not make the outer span safe:
state decides which partitions are selected, never how much of their span
counts, so under VALID_CLOSED_AND_DEGRADED the gateway reports
coverage_complete = true across a known gap and a request aimed at the gap
returns a successful empty result. Publishing the outer span is therefore
forbidden in every lifecycle state. Re-keying so each published partition is
contiguous is permitted but never mandated, because letting transport accidents
choose partition keys is an implementation leak. A catalog coverage relation is
the natural future evolution and is not available in v1. This ADR makes the
situation detectable and bounded, not resolved.
Producers gain an obligation: every eligible partition needs a coverage manifest before it can be published. There is no historical data to migrate, because no partition has been published through a bridge that does not yet exist.
Fixing the reference validator's timestamp parser to keep nanoseconds was
required by this contract: datetime.fromisoformat truncates below the
microsecond, and observed event bounds (first_exchange_ts/last_exchange_ts,
frozen at nine fractional digits) must compare exactly for the I8 containment
check to be trustworthy. The validator now matches epoch_ns throughout.
This ADR does not implement the bridge, a collector, live ingest, storage tiering, or any change to DataGateway.
Addendum — independent review hardening¶
An independent review of this candidate raised four blockers and five important findings, all accepted and closed without reopening the core decision above. In summary:
- the supersession graph (
coverage_id/supersedes) is now validated for self-reference, cycles, branching and dangling targets, fail-closed: any structural defect makes the whole reconstruction call publish nothing, rather than silently dropping the affected documents; - a
completeassertion must now resolve to exactly one existing, eligible partition manifest, and apartition_keymay have at most one live (non-superseded) revision among the manifests supplied; - declared coverage boundaries (not observed event timestamps, which stay
nanosecond-capable and frozen) are capped at microsecond precision, matching
what
catalog.partitions.ts_start/ts_end— PostgreSQLtimestamptz— can hold exactly, closing a mismatch the original nanosecond-parser fix had left open for the declared side; row_count == 0now mechanically requires null observed bounds, closing a direction the frozen schema's own conditional leaves open;- every assertion carries a document-scoped
assertion_id, andreconstruct_catalog_coverage()refuses mixed-DatasetIdentityinput outright.
Full rationale is in DECLARED_COVERAGE.md invariants I1, I3, I10 and I13–I15, and executable proof is in test_coverage_lineage_and_reconstruction.py.