ADR-0023 — Producer–Consumer Conformity Gate v1¶
Status: ACCEPTED
Contract Freeze Gate: PASSED
Conformity Implementation Gate: PASSED
This ADR records the accepted governance and semantic boundary. Runtime
implementation progress is tracked by SCOPE.md,
docs/product/ROADMAP.md and docs/product/CAPABILITY_MAP.md; this ADR does
not serve as the runtime implementation ledger.
Date: 2026-08-30
Context¶
The Data Plane producer track (ROADMAP.md "Parallel producer track") and the consumer dependency track (DataGateway -> Representation -> Feature -> ... -> API -> Clients) have so far been documented as independent parallel tracks meeting only "through the existing published Data Plane contracts and the DataGateway boundary."
An independent audit of the pre-implementation baseline recorded for this ADR — ADR-0019
/ DATA_GATEWAY.md, ADR-0021
/ CANDLE_DEFINITION.md,
ADR-0022 /
DECLARED_COVERAGE.md, and the narrow
DataGateway v1 implementation in src/quant_platform/data/ — found that the
seam where the producer track's output becomes the consumer track's input is
not yet precise enough to implement against without inventing semantics. Eight
findings (B1–B8, below) were accepted as gate-blocking, verified against the
actual repository state:
tools/import_bybit_trades.pywrites JSONL (write_jsonl_atomic), not Parquet; no canonicaltrade-v1Parquet producer exists (B1).- Commit
a166179("producer/manifest-catalog-bridge-v1") added only thecoverage-manifest-v1contract, schema, fixtures and ADR-0022; no bridge code exists insrc/ortools/(B2). - DATA_GATEWAY.md §8 states that a
validpartition is trusted without a full content rehash, but no document defines the evidence that must exist before a partition is markedvalid(B3). DataGateway.read()insrc/quant_platform/data/gateway.pyaccumulates every partition's rows into one Pythonlist, extends it across partitions, and performs one globalrecords.sort(...);read_trade_v1insrc/quant_platform/data/parquet.pymaterializes a fullpyarrow.Tableviascanner.to_table()per file. Both are O(total requested rows) memory (B4).- The determinism that global sort currently provides is not backed by any documented producer obligation about physical row order, so a future streaming rewrite that removes the buffering has nothing to guarantee ordering in its place (B5).
returned_boundsingateway.pyis constructed as aCoverageInterval— the same type used for declared/eligible coverage — reusing a coverage concept for a returned-data-only concept, and no finite-stream lifecycle (open/reading/completed/aborted) is defined anywhere (B6)._metadata()ingateway.pyincludescontent_hashesin the payload hashed intoresult_identity, exactly as DATA_GATEWAY.md §5 specifies. This is not a bug against the current frozen text, but it couplesresult_identityto physical artifact bytes in a way that conflicts with ADR-0021's materialization-equivalence principle ("recomputing the same semantic result ... does not create a new semantic materialization merely because the file, compression, catalog row or process changed") once a producer can regenerate byte-different Parquet for identical canonical records (B7).DataRequest.ordering_policydefaults to"bybit-trade-v1-exchange-ts-trade-id-v1"(models.py) andgateway.py._validate_supportrequires it verbatim, whilefixtures/candle-definition-v1/golden-5m.jsonfreezessource.ordering_policy: "trades@1-canonical-total-order-v1"as part of the CandleDefinition v1 identity hash. No document states how the two relate (B8).
Left unresolved, a producer-side implementation mandate (Parquet writer,
publication bridge, certifier) and a consumer-side implementation mandate
(streaming DataGateway, RecordTimeBounds, Candle runtime) would each have
to invent the other side's half of the seam, with no guarantee the two
inventions would agree.
Second review round¶
A first candidate of this ADR and
PRODUCER_CONSUMER_CONFORMITY.md
was independently reviewed and returned REQUEST CHANGES. That review found:
- B9 (new) — the frozen Bybit deterministic ordering policy,
(exchange_ts, trade_id), is unenforceable iftrade_idis null, buttrade-v1.jsonlegitimately allowstrade_id = nullfor sources with no native trade identity. The first candidate did not separate generic record-schema validity from Bybit publication eligibility. (Raised by the reviewer under the label "B1"; renumbered B9 here to avoid colliding with the original B1, which remains a distinct, still-valid finding about the absent Parquet producer.) - B2 refinement needed — the bridge invariants (BC1–BC8) did not define semantic catalog-rebuild equality (PostgreSQL UUIDs are not reproducible identities) or where durable certification evidence physically lives.
- B7 refinement needed —
CanonicalContentHashwas conceptually defined (§9 of the first candidate) but not specified as a reproducible algorithm: no domain separation, field order, serialization or framing was frozen. - Structural blocker — the first candidate's single "gate PASS" concept was circular: Decision §1 suspended all runtime until gate PASS, while Decision §6's exit criteria implicitly required tested runtime slices to exist. A gate cannot require its own postcondition as its precondition.
- Precision defects — the durable certification-evidence persistence
target was not named; the ordering proof (§7/OR4 of the first candidate)
over-attributed cross-partition non-overlap to declared-coverage contiguity
alone rather than to DataGateway's existing fail-closed
CatalogConflictcheck; the frozen-ADR test guard omitted ADR-0021; Decision §1 referenced "§10 below" for exit criteria that were actually in a different item; and the contract's own test suite mixed contract-consistency checks with language that could be read as claiming behavioral proof.
This revision resolves all of the above. It does not reopen B1–B8's original dispositions, which are unchanged.
Third review round¶
The revised candidate was independently re-reviewed and returned
REQUEST CHANGES once more, with one remaining blocker and two important
corrections, all resolved in this revision:
- Blocker — the durable certification evidence model (§13.3 of the
conformity contract) required a
quality_reportsrow withstatus='pass'to authorizestate='valid', but did not require that row to carry a non-null certifier identity. Astatus='pass'report with a null or emptycode_refcould therefore have authorizedvalidwith no record of which certifier implementation produced the evidence. Resolved by new invariants CE6 (mandatory, non-null, non-emptyquality_reports.code_refas an independent precondition alongsidestatus='pass', using the existing column, not a new one) and CE7 (certifiercode_refand producercode_refare distinct roles, never conflated). - Important — a JSON example in §13.3 used
"revision": 0, contradictingNaturalPartitionIdentity's own positive-revision requirement (DATA_GATEWAY.md §3,partitions.revision integer NOT NULL DEFAULT 1 CHECK (revision > 0)). Corrected to"revision": 1. - Important — the frozen
trade-v1.jsonguard test computed a SHA-256 digest and asserted only that it was 64 hexadecimal characters long, which passes for any file and proves nothing. Replaced with a canonical-JSON digest pinned against an explicit expected value, computed from the schema content rather than raw file bytes, so it is not sensitive to line-ending conversion.
This revision does not reopen B1–B9's dispositions or the two-gate model, both unchanged.
Decision¶
1. Two gates, not one — resolving the circularity¶
Two distinct, independently exitable gates are declared, so that "runtime is suspended until the gate passes" and "the gate's exit criteria require tested runtime" are never the same claim about the same gate:
foundation contracts (ADR-0019, ADR-0020, ADR-0021, ADR-0022)
|
v
CONTRACT FREEZE GATE <- documentation-level; exit = Decision §6
exit authorizes: implementing the conformity slices
(Parquet writer, manifest/coverage emission, certifier, bridge, bounded
DataGateway read) against this frozen contract
|
v
[implementation work happens here — authorized by Decision §8 after Contract
Freeze Gate PASS]
|
v
CONFORMITY IMPLEMENTATION GATE <- runtime-level; exit = Decision §7
exit authorizes: resuming producer AND consumer vertical expansion, and
Candle runtime, per ROADMAP.md
Contract Freeze Gate — purpose: authorize implementation of the conformity slices named in PRODUCER_CONSUMER_CONFORMITY.md §23. Its exit criteria (Decision §6) are entirely documentation-level: independent review acceptance of this ADR and the conformity contract, with no unresolved shared semantic blocker. It does not require any runtime to exist. Passing it does not reopen broader producer or consumer vertical expansion — only the conformity slices themselves.
Conformity Implementation Gate — purpose: prove the shared seam in
executable reality. Its exit criteria (Decision §7) are entirely
runtime-level: the conformity slices are implemented and pass the required
tests, including the golden vertical end-to-end proof. Only passing this
gate reopens broader producer vertical expansion, broader consumer vertical
expansion, and Candle runtime, per ROADMAP.md.
No document in this repository may state both "no conformity runtime before gate PASS" and "gate PASS requires conformity runtime" about the same gate. Where prior drafts used one undifferentiated "gate PASS," every occurrence is now one of these two named gates.
2. The conformity boundary is frozen¶
As the composition of existing contracts — canonical records -> canonical
physical materialization -> DatasetManifest/PartitionManifest/CoverageManifest
-> certification -> catalog publication -> eligible DataGateway source ->
bounded finite historical read -> Representations/application services — not
a new canonical layer. Its normative content lives in
PRODUCER_CONSUMER_CONFORMITY.md.
3. Findings disposition¶
B1–B8 (original) and B9 (new, this revision) are the gate-blocking concerns. Each has an accepted interpretation; B5, B7 and B9 carry explicit refinements, recorded in PRODUCER_CONSUMER_CONFORMITY.md. No finding is resolved by silently reinterpreting ADR-0019, ADR-0021 or ADR-0022; where existing frozen meaning needs to change, this ADR states that explicitly (§4 below) rather than hiding it in prose.
4. This gate does not mutate any accepted ADR¶
ADR-0019 (DataGateway), ADR-0020 (Consumer API), ADR-0021 (CandleDefinition
v1) and ADR-0022 (Declared Coverage) remain accepted and unchanged. Where the
conformity contract needs new semantics beyond what they state, it introduces
them as additive extensions (for example, a canonical_content_hash_v1 field
alongside the existing result_identity, per
PRODUCER_CONSUMER_CONFORMITY.md
§12) — never as an in-place reinterpretation. ADR-0021 in particular is
unchanged even though CanonicalContentHashV1 (§11) and Candle ordering
compatibility (§8) both depend on it: they depend on its existing text, they
do not alter it.
5. Dependency ordering for the eventual runtime work¶
Frozen in
PRODUCER_CONSUMER_CONFORMITY.md
§23. In summary: shared semantic primitives (RecordTimeBounds,
CanonicalContentHashV1, ordering compatibility) come first; the physical
Parquet materializer and the bounded DataGateway read path may then proceed in
parallel; certification depends on the materializer; the catalog bridge
depends on certification; the golden end-to-end proof depends on all of the
above.
6. Contract Freeze Gate exit criteria (all required)¶
- the physical
trade-v1Parquet contract (§9 of the conformity contract) is frozen and matches the columnssrc/quant_platform/data/parquet.pyalready reads; - the Bybit first-vertical eligibility profile (§10, resolving B9) is frozen,
distinct from generic
trade-v1record-schema validity; - the bounded historical read lifecycle and
RecordTimeBoundscontract (§§5–6) are frozen and distinct fromCoverageInterval; - the ordering contract for bounded reads (§7) and the DataGateway/Candle
ordering-identity compatibility mapping (§8, resolving B8) are frozen, with
the cross-partition non-overlap proof correctly attributed to DataGateway's
fail-closed
CatalogConflictcheck, not to declared-coverage contiguity alone; CanonicalContentHashV1(§11, resolving B7) is frozen as a reproducible byte-exact algorithm, and its relationship toPhysicalArtifactHashand the unchangedresult_identity(§12) is frozen;- the first-vertical certification rule for
state=valid(§13, resolving B3), including the durable evidence persistence target and the certification/publication sequencing, is frozen; - the manifest+coverage-to-catalog mapping and semantic catalog-rebuild equality (§14, resolving B2) are frozen;
- the golden conformity acceptance target (§15) and its required future proof obligations are recorded;
docs/product/ROADMAP.md,docs/product/CAPABILITY_MAP.md,SCOPE.mdanddocs/architecture/OPEN_DECISIONS.mdconsistently use the two-gate model;- this ADR and the conformity contract have been independently reviewed and accepted, the same review discipline already applied to ADR-0019 through ADR-0022.
Passing the Contract Freeze Gate is a documentation-and-governance event. It authorizes implementation of the conformity slices; it does not itself certify that any future runtime implementation is correct, and it does not reopen broader producer or consumer vertical expansion.
7. Conformity Implementation Gate exit criteria (all required)¶
- the canonical
trade-v1Parquet materializer exists and satisfies §9 and §10 (physical contract and Bybit eligibility profile), including the canonical -> Parquet -> canonical round-trip test; - the bounded DataGateway read path exists and satisfies §5–§8 (bounded-read
property,
RecordTimeBounds, ordering, ordering-identity compatibility), demonstrated with a memory-bounded-growth test; CanonicalContentHashV1is implemented per §11 and demonstrated reproducible across at least two byte-different, semantically identical Parquet artifacts;- the certifier exists and enforces §13, with a negative test per evidence
category (§13.2) showing the missing-evidence case refuses
valid; - the manifest+coverage-to-catalog bridge exists and enforces §14, including a demonstrated catalog rebuild that satisfies §14.4's semantic equality definition;
- the golden Bybit BTCUSDT 2024-01-15 vertical (§15) passes end-to-end, including the memory-bounded-growth proof (§15, GV2);
- the required adversarial conformity tests (§15.3) pass: zero-event coverage, gap/non-contiguous refusal, supersession, overlap rejection, relocation, catalog rebuild, physical-layout variation, abort-before-completion;
- the CandleDefinition ordering-compatibility acceptance test (§8.3) passes.
8. After Contract Freeze Gate PASS¶
Implementation of the conformity slices — Parquet writer, manifest/coverage
emission, certifier, catalog bridge, bounded DataGateway read path — was
authorized against the frozen contract. Those slices and the Decision §7
acceptance evidence are now complete on origin/main; the resulting Gate
closure is recorded below without changing the frozen criteria.
Under this Decision, broader Producer/Consumer verticals and Candle runtime
remain suspended until the Conformity Implementation Gate passes. That
condition is now satisfied; Decision §9 governs their post-Gate progression
through explicit scopes.
9. After Conformity Implementation Gate PASS¶
Producer vertical development and consumer vertical development resume as independent parallel tracks against the proven shared seam, exactly as ROADMAP.md already intends, and Candle runtime implementation may resume per its roadmap position. This is a temporary integration gate pair, not a new permanent architecture layer: once both are passed, no future work needs to re-derive or re-prove the conformity contract to proceed, only to extend it through the normal ADR-0018 evolution process if a frozen piece needs to change.
Consequences¶
- A subsequent mandate to implement the Parquet writer, the publication
bridge, the certifier, or the bounded
DataGatewayread path can proceed directly from PRODUCER_CONSUMER_CONFORMITY.md once the Contract Freeze Gate passes, without re-deriving cross-boundary semantics. - The golden Bybit BTCUSDT 2024-01-15 vertical (rows = 1,105,145; buy =
553,875; sell = 551,270) passed the full source -> canonical Parquet ->
manifests -> certification -> catalog -> bounded DataGateway chain. Its
accepted evidence is recorded in
docs/integration/HUMAN_GOLDEN_E2E_BYBIT_BTCUSDT_2024-01-15.mdand PR #20. result_identityas defined by ADR-0019/DATA_GATEWAY.md §5 is unchanged and remains physical-evidence-sensitive;CanonicalContentHashV1is additive, not a replacement, so no existing provenance chain is invalidated.- A Bybit
trade-v1partition withtrade_id = nullremains schema-valid under the frozen, unchangedtrade-v1.json, but can never reachstate = validunder the Bybit first-vertical eligibility profile (§10 of the conformity contract) and therefore can never become DataGateway-eligible data for this vertical. - Declaring the Contract Freeze Gate suspends only vertical expansion beyond the conformity slices. It does not forbid unrelated maintenance, documentation or test work elsewhere in the repository, and — once passed — it does not forbid the conformity slice implementation work itself.
Review gate¶
The historical review gate kept this ADR in PROPOSED status and kept
PRODUCER_CONSUMER_CONFORMITY.md
proposed until independent review confirmed the Contract Freeze Gate criteria.
That review is complete and the Contract Freeze Gate has passed. The
subsequent runtime slices and required evidence satisfied all eight Decision
§7 criteria, and the final read-only Gate review returned APPROVE with no
blockers or important findings. The Conformity Implementation Gate has
therefore passed; Decision §9 now governs the post-Gate program order.
Acceptance provenance¶
The Contract Freeze Gate acceptance is recorded in merged PR #8,
f0d100773f8ce1515c7eca49e67d96ad321eb50d, which merged the
Producer–Consumer Conformity documentation after independent contract review.
The review record states:
- independent audit and multiple read-only re-reviews completed;
- final review result: APPROVE;
BLOCKERS: NONE;IMPORTANT: NONE;- Contract Freeze Gate: PASS;
- READY TO IMPLEMENT CONFORMITY SLICES.
This is repository provenance for the accepted governance state, not an additional semantic or implementation requirement.
Conformity Implementation Gate acceptance provenance¶
The Conformity Implementation Gate closed against Decision §7 with the following merged evidence chain:
- Human Golden E2E / PR #20 — full Bybit BTCUSDT
2024-01-15vertical PASS; - adversarial refusal / PR #21 — eligibility and gap refusal PASS;
- adversarial acceptance / PR #22 — zero-event coverage and supersession PASS;
- evidence completion / PR #23 — real overlap rejection, RB1 rebuild, physical layout variation and Candle ordering composition PASS;
- Final Conformity Gate Review at authoritative
main84a2705bc1ccc94c8d68cbdeb874b3312a2f78fc— APPROVE,BLOCKERS: NONE,IMPORTANT: NONE.
The final ledger records all eight Decision §7 exit criteria as PASS, Adversarial Acceptance A1–A9 as PASS, Candle Ordering Compatibility as PASS, zero missing implementation proofs and zero unresolved Gate dispositions. Accordingly:
DataSliceMetadata.canonical_content_hash remains an additive future
extension and is non-blocking for this Gate. Semantic relocation invariance is
proved; an operational relocation subsystem remains a post-Gate capability.