Skip to content

Market Data Ingest Contracts

Status: Contract foundation v1 — producer-side semantics; no new record schema

Related: implementing the Parquet writer, publication bridge and certifier described here requires the Contract Freeze Gate to pass first (ADR-0023); broader producer vertical expansion beyond these conformity slices remains suspended until the Conformity Implementation Gate passes. PRODUCER_CONSUMER_CONFORMITY.md freezes the physical trade-v1 Parquet contract (§9, filling this document's absent producer), the manifest+coverage-to-catalog bridge obligations (§14, filling §9's "missing producer bridge"), and the first-vertical valid certification evidence rule (§13, filling §10's open quality-to-lifecycle mapping for the Bybit trade-v1 day).

This document defines obligations at the producer boundary. It reuses the existing contracts rather than creating a second identity, provenance, quality, catalog, or partition model.

1. Authorities and non-duplication

The following remain authoritative:

No section below changes those authorities.

2. Producer obligation

Market Data Ingest must produce a partition that is:

  • represented by the existing natural DatasetIdentity and partition identity;
  • compatible with its versioned record schema;
  • traceable through manifest, content, lineage, producer, and code provenance;
  • classified with existing lifecycle vocabulary;
  • reconciled against known source and temporal evidence;
  • accompanied by quality and provenance evidence; DataGateway, not the producer, applies its own read-eligibility policy.

The producer must fail visibly on contradictory identity, invalid schema, unresolved source conflict, or fabricated source fact.

3. Source adapter obligations

An adapter must expose or preserve, where the source supplies them:

  • authentication and protocol boundaries;
  • source pagination or subscription behavior;
  • native venue/instrument identifiers;
  • exchange timestamp and precision;
  • trade/aggressor semantics;
  • native trade, sequence, and order identifiers;
  • snapshot versus incremental meaning;
  • source-specific validation and failure conditions;
  • recovery and backfill capabilities;
  • source provenance.

The adapter may translate source representation into an internal mapping, but must not silently infer unavailable values. It must not own feature, research, strategy, execution, or client semantics.

4. Acquisition invariants

Historical, backfill, and live modes may use different transports and runtime components, but they share these invariants:

  • the same market fact has compatible canonical meaning;
  • source timestamp precision is preserved;
  • source-native identity is preserved where available;
  • missing receive time, sequence, trade ID, order ID, or ordering guarantee is represented as missing;
  • the requested acquisition scope is explicit;
  • source coverage, gaps, overlaps, and limitations are recorded;
  • retry and restart do not silently create a second semantic dataset.

Historical and backfill are finite operations. Live is persistent acquisition; its reconnect, checkpoint, rate-limit, and health behavior is operational and does not define a new consumer API.

5. Canonicalization obligations

Canonicalization must:

  • target an explicit versioned record contract;
  • preserve exact decimal and timestamp semantics;
  • apply source mappings only when supported by source evidence;
  • preserve nullable source fields rather than fabricate replacements;
  • reject or quarantine invalid records according to an explicit policy;
  • be deterministic for the same source input and semantic configuration;
  • record the semantic transformation and execution code identity through the existing lineage and partition provenance model.

The frozen trade-v1 contract remains unchanged. L1, L2, and L3/MBO contracts require the ADR-0018 evolution process.

6. Observed event bounds versus declared coverage

These are different concepts.

Observed event bounds

partition-manifest-v1 fields first_exchange_ts and last_exchange_ts are the timestamps of actual first and last records. When row_count == 0, both are null. They must never be populated with interval boundaries.

Declared coverage

Declared coverage is the responsibility interval used to determine whether a DataGateway request is covered, including an interval containing no events. It must be deterministic, half-open where required by DataGateway semantics, and independent from observed event bounds.

The population rule for catalog.partitions.ts_start and ts_end is frozen by DECLARED_COVERAGE.md and ADR-0022: the values come from complete assertions in coverage-manifest-v1 and from nothing else. The dt=YYYY-MM-DD to UTC-day mapping remains something no component may assume; partition_key identifies a partition and does not carry an interval. The frozen partition manifest is unchanged; its stale wording is corrected by the errata in DECLARED_COVERAGE.md §7.

7. Zero-row semantics

Zero rows do not imply no coverage. A declared responsibility interval may be fully covered while its materialized partition has:

row_count = 0
first_exchange_ts = null
last_exchange_ts = null

The declared interval must come from a complete coverage assertion, not from synthetic event timestamps. DataGateway must distinguish:

no eligible coverage
        ≠
complete eligible coverage with zero events

The declared-coverage contract is DECLARED_COVERAGE.md; the frozen observed-bound fields remain truthful and are never an input to it.

8. Sealing and publication

The semantic publication sequence is:

`writing`
      ↓
sealed / `closed`
      ↓
manifest and catalog reconciliation
      ↓
quality and certification evidence
      ↓
`valid` / `degraded` / `invalid`

The invariant is that a consumer must not observe a partially materialized partition. This does not require an impossible transaction spanning a filesystem and PostgreSQL. It requires crash-safe staging, idempotent retry, natural-identity-keyed reconciliation, and recovery after interruption.

The producer emits the lifecycle classification and supporting evidence; it does not evaluate a caller's DataGateway lifecycle policy. DataGateway independently applies its own read-eligibility policy. writing is not default-eligible. closed is sealed but not certified. valid is default-eligible. degraded requires explicit policy. invalid and superseded are excluded. “Published” should be used only with a stated meaning such as catalog-registered or default-eligible.

9. Manifest-to-catalog handoff

The existing PostgreSQL catalog is the runtime query index. Durable manifests remain identity, provenance, rebuild, and reconciliation evidence.

The missing producer bridge must reconcile:

canonical artifact
+ dataset manifest
+ partition manifest
+ coverage manifest
        ↓
catalog dataset/partition/lineage metadata

It must reuse natural identity, partition revision, storage roots, hashes, lineage, producer, and code_ref, and it must take ts_start / ts_end from the coverage manifests as specified in DECLARED_COVERAGE.md §4. It must not introduce a second catalog or filesystem-derived consumer identity.

10. Quality and reconciliation

Quality evidence may include source integrity, temporal coverage, gaps, overlaps, duplicate identities, sequence continuity where guaranteed, snapshot consistency, schema validity, row and byte counts, checksums, and degradation reason.

Existing quality_reports, lifecycle states, manifests, catalog metadata, lineage, and provenance are the intended model. The exact quality-report to valid/degraded/invalid transition remains open.

Partition uniqueness, revision, supersession, and catalog constraints provide mechanical protection. They do not solve source precedence, overlap resolution, reconnect overlap, repair triggering, or deduplication when source identity is absent.

11. Concurrency and recovery

Current mechanical protections include:

  • unique natural dataset identity;
  • unique (dataset, partition_key, revision);
  • at most one non-superseded revision per partition key;
  • lifecycle exclusion of writing from normal reads;
  • manifest and content hashes;
  • catalog rebuild logging.

These are not a complete writer-coordination protocol. Leases, leader election, distributed locks, and a coordinator service are deliberately not defined here.

Open recovery cases include:

  • simultaneous live and backfill coverage;
  • interrupted canonical write or manifest write;
  • file present without catalog registration;
  • catalog row without file;
  • divergent manifest copies;
  • partial relocation;
  • retry after a process or host failure.

Any future repair may use revision N+1, validation, and supersession, but the algorithm and source precedence remain open.

12. Provenance minimum

The producer should preserve the existing stable chain:

natural dataset identity
  → record schema identity
  → natural partition identity and revision
  → manifest/content hashes
  → lineage transformation
  → producer and execution code reference

Source and acquisition details must be retained where necessary for reproducibility, without changing frozen v1 fields or creating a competing provenance system. If the current v1 manifests cannot represent a source fact, the representation requires an explicit future contract or migration-specific artifact rather than an invented value.

13. Non-responsibilities

Ingest contracts do not define:

  • FeatureDefinition or FeatureSetDefinition;
  • candles or derived features;
  • hypotheses, events, labels, or validation recipes;
  • strategy, execution, PnL, or model training;
  • API/client UI behavior;
  • live DataGateway transport or cursor semantics;
  • L1/L2/L3/MBO JSON schemas;
  • Docker topology or storage technology.

14. Open producer decisions

Declared coverage is resolved by ADR-0022 and DECLARED_COVERAGE.md. The following remain open:

  • publication of a partition whose declared coverage is not one contiguous complete interval;
  • physical placement of _coverage/ when one dataset spans storage tiers;
  • quality-to-lifecycle transition;
  • publication/reconciliation protocol details;
  • live/backfill source precedence and overlap policy;
  • deduplication without native identity;
  • second-venue deterministic ordering integration;
  • source provenance extension;
  • checkpoint and recovery semantics;
  • exact future L1/L2/L3/MBO contracts.