Skip to content

Consumer API Boundary Contract v1

Status: Contract v1 — semantic boundary frozen; transport/runtime not implemented

Decision: ADR-0020 — Consumer API semantic boundary

Scope: the canonical application boundary for App UI, TUI, CLI, notebooks, automation and other external consumers.

Related: PRODUCER_CONSUMER_CONFORMITY.md §16 clarifies, without changing any rule below, how a future bounded/streaming DataGateway read path composes with this contract's synchronous-read and future job semantics, and how RecordTimeBounds composes into returned_temporal_bounds.

This contract freezes the meaning and ownership of the consumer boundary. It does not implement a web framework, endpoint, client, job runtime or representation engine.

1. Roadmap position

The canonical consumer dependency track in ROADMAP.md places the product capabilities in dependency order:

DataGateway
  -> Representation foundation
  -> Feature foundation
  -> Research / Outcome
  -> Validation / Labeling
  -> Strategy / Decision
  -> Execution / Replay
  -> Experiment persistence
  -> Canonical API and job runtime
  -> Clients

The Data Plane producer track runs in parallel and meets this track through the published Data Plane contracts and the DataGateway boundary. It is not a second consumer boundary and does not change the ordering above.

A narrow catalog-backed DataGateway read slice is implemented; the DataGateway capability itself remains PARTIAL in the capability map, and broader access remains open. The representation and later domain capabilities remain ahead of API runtime implementation. This document is therefore a boundary-preparation milestone: it does not reorder the roadmap or claim that phase 9 runtime work has started.

The existing representation decisions are intentionally respected:

  • CandleDefinition owns reproducible candle semantics;
  • PARTIAL and CLOSED candles remain distinct;
  • future market-data contract extensions require explicit versioning;
  • unsupported representation fields are not invented here.

2. Purpose and exclusive application boundary

The canonical API is the only supported application boundary for:

  • App UI;
  • TUI;
  • CLI;
  • future notebooks and automation;
  • other external consumers.

The dependency direction is:

Consumers
    |
    v
Canonical API (semantic contract)
    |
    v
Application services / orchestration
    |
    +--> Representations / Feature Engine
    +--> Research / Outcome / Validation
    +--> Strategy / Execution / Experiments
    |
    v
Platform capabilities
    |
    v
DataGateway
    |
    v
Canonical data plane

The API is a semantic platform boundary, not a transport-shaped alias for DataGateway. An internal application service may use DataGateway, but the public contract must not be POST /data/read or an equivalent operation that requires callers to know gateway request fields, catalog selection, partition policy or physical storage.

The conceptual capability families are organized so the platform can grow without introducing a second client boundary:

Capability family Consumer-facing role in this contract
Market data Query a semantic representation over an explicit interval
Features Request versioned feature definitions/sets when implemented
Research and outcomes Submit or inspect declarative studies and outcomes
Labels and validation Request reproducible labeling/validation operations
Strategies and execution Submit decision, replay and execution intents when implemented
Experiments Inspect reproducible studies, runs and artifacts when implemented
Jobs Represent long-running operations across the capability families

Only the boundary principles and the first trades reference slice are contracted here. The other families are extension points, not implemented capabilities.

3. Three distinct concepts

3.1 Consumer query / application request

A consumer request expresses intent. The v1 conceptual market-data request is:

ConsumerMarketDataQuery v1
  selector
    venue: semantic venue identifier
    instrument: venue-native instrument identifier
  interval
    start: UTC instant
    end: UTC instant
    semantics: [start, end)
  representation
    kind: versioned representation kind
    version: representation semantic version
    definition: representation-owned parameters, only when supported
  options
    optional, capability-owned and versioned

The v1 selector is deliberately minimal: venue and instrument. It identifies what the consumer means, not a physical dataset, file name, path, catalog UUID or execution instruction. start == end is a valid empty interval; start > end is invalid. The interval is UTC and uses half-open semantics wherever the requested capability returns a finite historical slice.

Future selector dimensions such as market type, venue sets or cross-venue selection require an explicit versioned selector extension. Their semantics are not frozen by this contract.

The v1 reference representation is:

representation = { kind: trades, version: 1 }

The consumer-facing shorthand representation = trades denotes this versioned reference. It is not a dataset_kind or a request for a specific materialized dataset.

It requests canonical trade records for the selected venue and instrument. The consumer does not select trade-v1, a partition, a schema file or a storage location. The application service resolves those details against the available platform capabilities.

The request may contain a projection or other option only after the relevant capability has defined its semantics. Unsupported options fail explicitly; they are not silently ignored or passed through to a lower layer.

3.2 Representation definition

RepresentationDefinition is the semantic description of the form in which the consumer wants data. It is separate from both the query and the source dataset.

Examples of the extensible boundary are:

trades@1
candle(<versioned candle definition>)
footprint(<versioned footprint definition>)
book(<versioned book definition>)

Only trades@1 is a reference representation in this contract. Candle, footprint and book examples show where future definitions fit; they do not freeze fields, aggregation rules, L1/L2/L3 semantics, materialization policy or schemas for capabilities that are not yet implemented.

A representation definition has a stable semantic identity composed of its kind, version and canonical definition parameters once that representation's own contract is frozen. Its implementation may be computed on demand, served from a materialized artifact, read from cache or reconstructed from an approved canonical source. That choice is not part of consumer semantics.

3.3 Dataset identity

DatasetIdentity identifies available canonical or derived materialized data inside the platform. The frozen DataGateway Contract v1 defines the natural identity as:

(layer, dataset_kind, venue, instrument, record_schema_id)
  + (feature_set_slug, feature_set_version)  # only for feature datasets

It is not the primary consumer query model. The application service may map a semantic consumer query to one or more dataset identities, or to a representation computation that uses no directly materialized representation.

NaturalPartitionIdentity is likewise an internal source/provenance concept:

DatasetIdentity + (partition_key, revision)

The platform, not the consumer, decides whether a representation is backed by a materialized dataset, computed from canonical records, cached or served by another approved implementation.

4. Reference request and internal flow

The first reference slice is intentionally narrow but does not define the complete market-data model:

semantic consumer query
  selector: venue=Bybit, instrument=BTCUSDT
  interval: [start, end)
  representation: trades@1
          |
          v
canonical API boundary
          |
          v
market-data application service
  resolves the representation and semantic selector
          |
          v
internal DataRequest
  canonical natural DatasetIdentity
  schema/time/read policy chosen by the service
          |
          v
DataGateway
  catalog -> eligible partitions -> canonical records
          |
          v
canonical Bybit trade-v1 data

For the current reference implementation, the internal resolution is compatible with the canonical natural identity (canonical, trades, bybit, BTCUSDT, trade-v1). This is an implementation mapping, not a requirement that clients send that identity. A later candle(5m), footprint or book capability must be addable without changing the consumer's fundamental query boundary.

5. Results, coverage and provenance

The API result for a finite market-data query must preserve the meaning needed to interpret and reproduce the result without exposing physical mechanics. The exact row/columnar wire representation remains open until transport and return-shape work is scoped.

The stable result envelope is conceptually:

ConsumerMarketDataResult
  request
    normalized semantic query
    request_identity
  representation
    kind, version and canonical definition identity
  data
    logical records/batches in the representation's defined order
  interval
    requested_interval: [start, end)
    returned_temporal_bounds: first/last record time under the requested
      representation's defined temporal semantics, or null when empty
  coverage
    covered_intervals
    gaps
    complete: bool
  provenance
    source semantic identity and schema/version information
    stable source references where needed for reproducibility
    source/DataGateway coverage diagnostics, when useful
    implementation identity when computation or normalization occurred
  statistics
    row_count where meaningful

returned_temporal_bounds reports the first and last returned record time under the requested representation's own defined temporal semantics. Every representation contract must define the temporal coordinate needed to interpret those bounds. This contract does not define candle, footprint, book or L1/L2/L3 timestamp semantics; each representation owns them. For a successful empty result the bounds are null.

Coverage at the consumer boundary is representation/application-level coverage. covered_intervals describes where the requested representation is available under that representation's semantics; gaps describes the requested portions that are not covered; complete is a boolean that is true if and only if gaps is empty over the requested interval. A derived representation may have different coverage from its source because of its own definition, warmup, late-event or partial-result rules. Coverage is not a universal exposure of DataGateway eligible_coverage.

The application service may retain source/DataGateway coverage diagnostics in provenance. Those diagnostics are explanatory source evidence, not the stable consumer coverage abstraction. For the current trades@1 reference, the service preserves the frozen DataGateway v1 strict behavior when determining whether a semantic result can be produced:

full representation coverage + zero records -> success
                                               complete = true
                                               returned_temporal_bounds = null
no representation coverage                  -> no_coverage
representation gap under strict policy      -> no_coverage

The API may translate source outcomes into stable API error codes, but it must preserve the distinction between a complete empty result and unavailable or incomplete representation coverage. It must also preserve deterministic ordering and the representation's temporal availability semantics. It must not fabricate, interpolate or silently truncate uncovered data.

Provenance exposed at this boundary is semantic and diagnostic: source venue, instrument, representation/schema versions, coverage and stable source references when required for reproducibility. Consumers must not need to construct or persist catalog locators or physical paths to use the result.

6. Identity visibility matrix

Identity or locator Consumer query Stable API result/provenance Internal/runtime policy
request_identity Not supplied as a locator; generated from normalized request Visible and stable Must exclude runtime/catalog/path data
representation_identity Supplied semantically as kind/version/definition Visible when the representation is defined Changes when representation semantics change
result_identity Not required to request data May be returned when deterministic result identity is defined Derived from stable request/source/result semantics
Natural DatasetIdentity Not primary; never required for ordinary consumer requests May appear as provenance/diagnostic source reference Owned by DataGateway/data plane; not a client selection API
NaturalPartitionIdentity Never a consumer selection instruction May appear in reproducibility provenance where needed Internal source identity; not a physical locator
Catalog dataset/partition UUID Never Not part of stable API identity; diagnostic exposure is not frozen here Runtime/catalog locator only
storage_root_id, rel_path, absolute path Never Never part of stable API semantics DataGateway-only physical resolution

Catalog UUIDs, storage roots, relative paths and resolved filesystem paths must not affect request identity, result identity or semantic compatibility.

7. Error boundary and translation policy

The API exposes stable, structured errors rather than raw Python exceptions, SQL errors, filesystem messages or paths. Each error includes a safe logical context and, where applicable, the request identity; it does not expose an absolute path or internal stack trace as caller semantics.

The minimum market-data error vocabulary is:

API error code Meaning and required distinction
invalid_request Malformed interval, selector, unsupported option shape or contradictory request
unsupported_representation The requested representation kind/version is not implemented or available
source_not_found No approved source/dataset can satisfy the semantic request
no_coverage The requested representation has no covered interval or has an incomplete gap under its policy
schema_incompatible Available source/representation schema cannot satisfy the requested semantics
integrity_failure The platform cannot safely produce a trustworthy result

full representation coverage + zero records is a successful empty result, not no_coverage. The API may add capability-specific errors later, but it may not collapse these distinctions into a generic not-found or empty response.

Internal causes such as an unreadable partition state, catalog/manifests conflict or corrupt content remain diagnosable through safe internal details, tracing and operational logs. They are not frozen as consumer API error codes; the application service maps them to source_not_found or integrity_failure according to whether the consumer can act on source absence or only on platform failure.

8. Synchronous reads and asynchronous jobs

The API contract distinguishes operation class from transport:

  • A cheap, bounded, deterministic read may complete synchronously and return its result directly.
  • A long-running computation, materialization, research run, backtest, training operation or other operation whose duration/resource use is not bounded for an interactive request uses asynchronous Job semantics.

The future job contract must provide a stable job identity, lifecycle state, result/error reference, and cancellation/retry semantics where the operation supports them. Progress is reported only when it has defined meaning. A client must not emulate a job by holding an unbounded synchronous request open.

This document does not implement job storage, scheduling, workers, cancellation or progress reporting. It also does not require every future market-data representation to be asynchronous; the application service owns the boundedness decision under the capability's contract.

8.1 Non-loopback J02 security

The current loopback J02 listener remains a local-only deployment. Any future non-loopback J02 listener is governed by ADR-0063: it must use TLS 1.3 mutual TLS, validate the server endpoint and client certificate, and map the exact client credential fingerprint to an explicit principal and scope before decoding a Consumer API request. The only scope defined now is j02.market_data.read; it grants no canonical storage, catalog, checkpoint, backup, publication, Job, or administrative authority.

Authentication and authorization failure occur outside the Consumer API result envelope and must not reinterpret a ConsumerErrorCode. A remote listener may not fall back to plaintext or an unauthenticated WebSocket session. This is a future implementation contract; it does not change current loopback behavior or add a remote endpoint.

9. Versioning and evolution

The following version domains remain distinct:

  • API contract version: versioned semantic boundary and error/result rules;
  • representation version: versioned meaning and parameters of a requested representation;
  • dataset/schema version: canonical source and record meaning owned by the data plane/DataGateway;
  • runtime identity: catalog UUIDs, paths and process metadata, never a semantic compatibility key.

Semantic change to a frozen API contract requires an explicit new API version. Semantic change to a frozen representation requires an explicit new representation version. An ADR may authorize or document the migration, compatibility plan and version introduction, but an ADR alone cannot mutate frozen semantics in place. A source materialization can change from computed to cached or persisted without a consumer contract change if the representation semantics, provenance and result guarantees are unchanged.

No legacy API compatibility is promised. Existing ml_core behavior is read-only evidence and does not constrain this boundary. Compatibility with the frozen trade-v1, dataset-manifest-v1 and partition-manifest-v1 contracts cannot be obtained by silently changing those contracts; a future incompatible data meaning requires explicit contract evolution.

Transport versioning, authentication, pagination, streaming/cursors, snapshot publication, exact serialization and schema-v2 compatibility remain deferred decisions for their relevant implementation scopes.

10. Client invariants

App UI, TUI and CLI are equivalent clients of the same canonical API. Clients may own presentation, interaction state, local preferences and rendering.

Clients must not own:

  • market-data reconstruction or aggregation;
  • catalog or physical dataset discovery;
  • feature calculations;
  • label or outcome semantics;
  • strategy logic;
  • execution simulation;
  • retry logic that changes domain meaning.

The API/application-service layer owns request normalization, capability resolution, temporal semantics, provenance, deterministic ordering and domain logic. A client may compose a semantic request and render the result, but it must not bypass the API to invoke DataGateway, PostgreSQL, Parquet, a Representation Engine or another platform internal.

11. Contract invariants and explicit non-goals

This contract is accepted only if all of the following remain true:

  • consumers never access DataGateway directly;
  • consumers never resolve PostgreSQL, Parquet, storage roots or paths;
  • the API is not defined as a DataGateway HTTP wrapper;
  • trade-v1 is a reference implementation slice, not the general market-data model;
  • consumer market-data requests are semantic;
  • RepresentationDefinition and DatasetIdentity are separate concepts;
  • materialization strategy may change without changing consumer semantics;
  • future candle, footprint, book and other representations fit the same boundary;
  • runtime UUID/path identities do not leak into stable consumer semantics;
  • coverage, determinism and provenance remain meaningful through the API;
  • long-running operations have a future asynchronous Job path;
  • App, TUI and CLI share one canonical platform API;
  • quantitative business logic is not moved into clients.

Explicit non-goals for this milestone are FastAPI, React, TUI, CLI commands, candle/footprint/book builders, L1/L2/L3 implementation, Feature Engine, Research Engine, Job Runtime, authentication, deployment, schema migration, DataGateway generalization and legacy CoreApp/API porting.

12. Domain capability seam extensions

ADR-0065 defines the accepted application seam discipline for the future Strategy, Replay, Validation, and Training Consumer-API capabilities. Requests carry only existing immutable semantic payloads and content identities; paths, catalog UUIDs, repository handles, callable import names, and client-selected locators are never consumer semantics. Application owns normalization, governed resolution, invocation, and stable error translation; transport and clients do not reinterpret domain behavior.

The accepted J10 Strategy seam is compose_decision; J12 consists only of its existing finite fold/classification/DSR/PBO operations; and J13 is the existing supervised evaluation/owned-registration boundary. J11 Replay remains deferred: HistoricalReplayRuntime requires a feature_provider callable that has no accepted semantic, serializable reference. A later ADR must supply that authority before any Replay consumer request can exist. Long-running work is admitted through J03, never emulated by a client-held synchronous request.

13. Additive transport evolution

ADR-0066 keeps j02-request-v1/j02-response-v1 as one complete exchange, including their typed RESULT_TOO_LARGE refusal. A later J14 finite-result family may frame only an already-produced complete result with a declared logical result identity, deterministic chunk indices, per-chunk and complete payload digests, and a transfer-local resume token. The result bytes and the named transfer-id preimage use RFC 8785 JSON Canonicalization Scheme encoded as UTF-8; SHA-256 digests are lowercase hexadecimal. Frames are never application-level pages or a cursor for a new query.

No live Consumer message is defined by that transport decision. Any future live family must first have application-owned semantics and, when carrying B06, serialize ADR-0047's LiveStreamCursorV1 and event vocabulary unchanged. A finite transfer id must never be treated as a B06 cursor.

14. Omega J15 remote client contract

ADR-0067 freezes Omega J15 as a remote J02 consumer only. Its initial operation sends j02-request-v1 and accepts j02-response-v1 for the C03 market-data read admitted by j02.market_data.read. The supported public Python API and ADR-0055's local validation bridge are package dependencies, not an alternate remote-client boundary: no local fallback, mixed execution, wire-family downgrade, or remote compatibility decision based on quant_platform.__version__ or PLATFORM_PIN is permitted.

An unexpected J02 version or unadmitted message family is a local typed wire_incompatible failure, distinct from a losslessly carried ConsumerApiError. TLS, mTLS, and scope denial are a local typed remote_security_failure before Consumer API processing; a J15 client cannot fall back to plaintext or infer an unavailable capability. Omega renders server-issued request identities, provenance, coverage, results, and errors; it never owns canonical data, catalog state, or accepted-artifact identity.