CandleDefinition v1 Contract¶
Status: Contract v1 — semantic contract frozen; historical candle runtime not implemented
Decision: ADR-0021 — CandleDefinition v1 semantic contract
Related: historical candle runtime implementation is postponed until the
Producer–Consumer Conformity Gate v1's
Conformity Implementation Gate passes (docs/product/ROADMAP.md).
PRODUCER_CONSUMER_CONFORMITY.md freezes,
without changing any rule below: the compatibility mapping between this
contract's source.ordering_policy and DataGateway's ordering identity (§8,
including the Bybit eligibility profile in §10 that mapping now depends on),
and how a future CanonicalContentHashV1 composes with this contract's
materialization-equivalence alternative evidence paths (§2, §12).
Scope: fixed-duration, UTC epoch-aligned candles built from canonical
trades@1 / trade-v1 records
This is a representation contract candidate. It defines semantics before a runtime exists. It does not implement an aggregator, live collector, API endpoint, catalog migration or a general Representation framework.
1. Authority and v1 boundary¶
Candles are first-class derived representations, not features or UI objects.
CandleDefinition owns the reproducible meaning of one candle series. This v1
contract freezes only fixed-duration time candles over canonical trades. It does
not freeze footprint, book, volume-bar, tick-bar, L1/L2/L3 or calendar/session
representations. Those require their own explicit versioned contracts.
The first implementation slice is:
canonical trades@1 / trade-v1
-> CandleDefinition v1
-> fixed-duration UTC candles
-> CLOSED candle records
The source must already be a semantically valid canonical trade result. The candle layer does not become a second DataGateway, source deduplicator or physical-file reader.
2. Four distinct identity concepts¶
These concepts must not be collapsed:
| Concept | Owns | Does not own |
|---|---|---|
ConsumerMarketDataQuery |
semantic selector (venue, instrument), requested [start,end) interval and requested representation |
candle parameters, catalog UUIDs, paths or materialization choice |
CandleDefinition |
the versioned meaning of the candle representation and its parameters | venue, instrument, query interval, source dataset UUID, partition locator or runtime state |
DatasetIdentity |
a logical canonical or derived dataset, including its venue/instrument and data-plane schema identity | candle aggregation meaning or physical path as identity |
| materialization identity | one derived result/materialization tied to a definition, source snapshot/revision and support interval | storage root, relative path, compression, catalog UUID or process-local cache key |
venue and instrument therefore remain in the consumer semantic selector and
in the resolved source DatasetIdentity; they are not duplicated in
CandleDefinition. The same definition can be applied to different selected
instruments when their source satisfies the trades@1 dependency.
The conceptual semantic materialization identity is:
CandleMaterializationIdentity =
(CandleDefinition identity,
source snapshot/result identity,
ordered source partition identities and content revisions,
required bucket-support interval(s),
output record schema identity,
canonical result/content identity)
The source snapshot/result identity and the ordered partition/content tuple are
alternative evidence paths only when the snapshot identity itself is durable;
at least one complete immutable source-evidence path is required. This tuple
is not the CandleDefinition identity and contains no physical locator.
Recomputing the same semantic result from the same source evidence does not
create a new semantic materialization merely because the file, compression,
catalog row or process changed.
3. CandleDefinition semantic identity¶
3.1 Exact identity fields¶
The semantic identity of CandleDefinition v1 is the canonical serialization
of exactly these fields:
representation_kind
definition_version
source
duration_ns
alignment
boundary
aggregation
empty_bucket_policy
closure
late_event_policy
availability
output_record_schema
numerical_semantics
No field is inferred from a filename, UI default, source path, venue, instrument, query interval, catalog identifier, partition locator, process, cache, materialization or implementation language. V1 has no optional semantic fields and no aliases with different spellings or meanings.
The canonical payload for a five-minute definition is structurally:
{"aggregation":{"close":"last_by_source_order","high":"max_exact_decimal_price","low":"min_exact_decimal_price","open":"first_by_source_order","trade_count":"count_source_trades","volume":"sum_exact_decimal_size"},"alignment":{"kind":"utc_epoch","offset_ns":"0"},"availability":{"causal_floor":"bucket_end","observed_available_at":"nullable_observed_source_finalization_time","partial_consumption":"forbidden"},"boundary":{"bucket":"[start,end)","query_selection":"bucket_intersects_[query_start,query_end)","timestamp_coordinate":"bucket_start"},"closure":{"closed_state":"CLOSED","evidence":"source_finalization_required","partial_state":"PARTIAL"},"definition_version":1,"duration_ns":"300000000000","empty_bucket_policy":"omit","late_event_policy":"revisioned_rebuild","numerical_semantics":"exact_decimal_no_rounding_v1","output_record_schema":"candle-v1","representation_kind":"candle","source":{"event_time_field":"exchange_ts","ordering_policy":"trades@1-canonical-total-order-v1","record_schema":"trade-v1","representation":"trades@1"}}
The one-line form above is illustrative of the bytes, not a permission to
omit fields. Object keys are sorted recursively, UTF-8 is used, whitespace is
omitted, and the payload is serialized using RFC 8785 JSON Canonicalization
Scheme rules. duration_ns and offset_ns are strings specifically to avoid
JSON number precision loss. The definition identity is:
The hash is a semantic identity, not a materialization or runtime identity. Changing any listed field requires a new representation/definition version; accepted v1 meaning is never silently reinterpreted.
3.2 Duration canonicalization¶
The semantic duration is a positive integer number of nanoseconds, serialized
as a decimal digit string with no leading zeroes ("1", not "01"). It is
not a floating-point value and cannot use calendar arithmetic.
A surface parser may accept only these explicitly defined forms:
- an unsigned integer followed by one of
ns,us,ms,s,m,h,d; HH:MM:SSwith an optional fractional second of one to nine digits.
The unit multipliers are fixed SI durations; d means exactly 86,400 seconds,
not a local or calendar day. The colon form has 0 <= MM < 60 and
0 <= SS < 60. All forms are converted to the one canonical duration_ns
field before identity is calculated. Thus 5m, 300s and 00:05:00, when
accepted by the parser, all become "300000000000". Any other spelling is
rejected rather than assigned an accidental meaning. V1 does not support
months, calendar periods, sessions or exchange-local durations.
3.3 Source representation dependency¶
The v1 source object is exactly the semantic dependency on canonical
trades@1 backed by trade-v1:
- event time is
trade-v1.exchange_ts; - price is
trade-v1.price; - size is
trade-v1.size; - source ordering is the deterministic total order guaranteed by the
trades@1result under the namedtrades@1-canonical-total-order-v1policy; - no filesystem/batch/reader iteration order is valid.
For the current Bybit reference slice, that deterministic order is
(exchange_ts, trade_id) using parsed timestamp order and the source's
canonical opaque-string lexicographic trade-id comparator. A trade ID is
not parsed as a number: values such as "100", "20", "9" retain the
source ordering "100", "20", "9". A venue sequence, when it is the
declared source tie-breaker in a future source policy, is compared by its
numeric value, never lexicographically as its serialized string. A source
result without a deterministic total order for all records in a bucket is not
a valid v1 candle input and must fail explicitly. The candle definition does
not invent a tie-breaker.
receive_ts is not required for the historical source. If it is absent in
trade-v1, it remains absent; it is never derived from exchange_ts, a file
name, import time or row position.
4. Alignment, boundaries and temporal coordinates¶
V1 alignment is UTC Unix-epoch alignment only. The reference is
1970-01-01T00:00:00Z; the offset is fixed to 0 nanoseconds. Local time,
timezone offsets and DST never participate. Offset alignment is not a v1
feature and a non-zero offset is rejected.
For an event timestamp t and duration D, the bucket start is calculated
using mathematical floor division:
The calculation is defined over integer nanoseconds, including for timestamps before 1970; truncation toward zero is not equivalent and is not permitted. Each bucket is half-open:
A trade exactly at bucket_start belongs to that bucket. A trade exactly at
bucket_end belongs to the next bucket. The candle's canonical temporal
coordinate and record time are bucket_start; bucket_end is also present in
the record so the full interval is explicit. Neither coordinate is an
availability timestamp.
5. Deterministic OHLCV and count semantics¶
For each non-empty bucket, source trades are consumed in the deterministic source order described in section 3.3:
openis the price of the first source-ordered trade;highis the exact-decimal maximum of all trade prices;lowis the exact-decimal minimum of all trade prices;closeis the price of the last source-ordered trade;volumeis the exact-decimal sum of all trade sizes;trade_countis the count of source trade records in the bucket.
aggressor_side does not change OHLCV in v1. No VWAP, delta, buy/sell
volume, imbalance, intensity or other derived feature is silently added.
Price comparison and size addition use arbitrary-precision exact decimal semantics. Canonical aggregation must not parse, compare, sum or round these values through binary floating point. V1 has no implicit scale, tick-size rounding, clamping or overflow fallback; an implementation that cannot retain the exact value must fail rather than round it.
Source decimal strings retain their source representation under trade-v1.
Output decimal strings use one canonical numeric spelling: no exponent, no
leading zeroes, and no trailing fractional zeroes; zero would be "0".
All v1 OHLC and volume values are strictly positive, so zero is not valid for
those fields. trade_count is a canonical non-zero digit string because a
non-empty candle always has at least one source trade and counts must not be
exposed to JSON/float precision loss.
6. Empty buckets¶
An otherwise fully source-covered bucket with zero trades produces no candle record. V1 does not emit null OHLC, zero OHLCV, or carry-forward values.
This omission is not a coverage gap. Coverage is established from declared source support and finalization evidence, not from returned row count or observed event bounds. Therefore:
full source support + zero trades -> no row, covered bucket
missing source support -> uncovered bucket / no CLOSED result
An empty result over a fully supported requested bucket set is a successful,
complete empty result with null returned record bounds at the consumer
envelope. Empty candle buckets do not appear in candle-v1 because there is no
reproducible OHLC record to serialize.
7. Query interval, bucket support and returned records¶
The consumer query interval remains [query_start, query_end). For a candle
representation, a candidate bucket is selected when its full interval
intersects the requested interval:
if query_start == query_end:
candidate_buckets = empty
required_bucket_support = empty
returned records = empty
representation coverage is complete over the empty requested interval
else:
candidate bucket iff:
bucket_start < query_end
AND
bucket_end > query_start
query_start > query_end is invalid under the frozen Consumer API contract.
The equality case is an explicit exception to the overlap predicate above;
it is not evaluated as an ordinary non-empty interval.
Only complete candidate buckets are constructed. Only non-empty candidate
buckets are returned. The returned candle is never truncated to the query
edge; its full [bucket_start,bucket_end) remains explicit.
For example, with D = 5 minutes and request [10:02, 10:07):
constructed buckets: [10:00, 10:05), [10:05, 10:10)
returned coordinates: 10:00 and 10:05, for non-empty buckets
source-support interval: [10:00, 10:10)
Both boundary buckets are selected because they intersect the request. The
first selected bucket requires support before query_start and the last can
require support after query_end; these are explicit representation-support
rules, not hidden lookahead or feature warmup. Any non-empty query therefore
has at least one candidate aligned bucket. A zero-length query has no
candidates, including both [10:02,10:02) and [10:00,10:00); each has empty
required support, empty returned records and complete representation coverage
over its empty requested interval.
required_bucket_support is the union of complete [bucket_start,bucket_end)
intervals for all candidate buckets. It is the interval(s) the source must
cover to construct the requested representation. It is distinct from both the
consumer query interval and the returned record coordinates.
The same rule applies to on-demand and materialized reads. A materialization may store a larger support range, but the consumer still receives only candidate records whose bucket intervals intersect the query interval.
8. Coverage and precise terminology¶
Representation coverage is not identical to DataGateway/source coverage:
declared source coverage
-> required bucket support
-> finalizable candle buckets
-> returned non-empty candle records
For v1 historical construction, every required bucket-support interval must
be fully covered by eligible source coverage, with no internal gap, and must
have source finalization evidence. The source read uses the existing strict
DataGateway coverage semantics; an uncovered support interval is NoCoverage
and must not yield a partial series or a falsely complete result.
Full source support plus zero events is still complete coverage. Observed
first_exchange_ts/last_exchange_ts bounds are not declared support and
cannot be substituted for it. Incomplete first/last source support does not
become a partial candle; the bucket is not CLOSED.
To compose with the frozen Consumer API coverage envelope, project each
candidate bucket interval onto the requested interval. A fully supported and
finalizable bucket contributes its intersection with the request to
coverage.covered_intervals; an incomplete candidate contributes that same
intersection to coverage.gaps. The union is normalized using the same UTC
half-open rules as the Consumer API. coverage.complete is true exactly when
the projected gaps are empty. For [10:02,10:07), complete support for both
candidate buckets reports covered [10:02,10:07) even though source support is
[10:00,10:10). If [10:05,10:10) lacks source support, the requested gap is
[10:05,10:07) and strict v1 returns NoCoverage; it does not return the
other bucket as a silently partial series. For [10:02,10:03), the
intersecting [10:00,10:05) bucket projects to covered [10:02,10:03);
there is no requirement for an aligned bucket_start inside the query. A
fully supported empty bucket contributes coverage even though it contributes
no record.
There is no indicator warmup in CandleDefinition v1. The source interval
needed to complete aligned buckets is called required bucket support or
support extension, never feature warmup. A support extension can be on
both sides of a non-aligned non-empty query in v1 because full intersecting
buckets are required. For [10:02,10:07), support extends left to 10:00 and
right to 10:10.
9. PARTIAL and CLOSED state model¶
State belongs to a runtime observation/envelope, not to the canonical
candle-v1 record. A materialized candle series contains only CLOSED
non-empty records plus its coverage/provenance envelope.
PARTIAL¶
PARTIAL is mutable, incremental and unsealed. It may be updated as source
events arrive or as source completeness changes. It is not a reproducible
historical result, must not be consumed as a closed candle by Features,
Research or Strategy, and is not eligible for canonical closed-series
materialization.
CLOSED¶
CLOSED is immutable under its source snapshot/revision. A non-empty CLOSED
record is eligible for reproducible historical use and materialization. An
empty finalized bucket is represented by coverage/finalization evidence and
absence of a row, not by a fabricated CLOSED record.
Closure requires all of the following:
- complete eligible source coverage for the entire bucket support interval;
- a deterministic source ordering contract;
- a stable source snapshot/revision and content evidence;
- explicit source finalization/completeness evidence for that interval.
Wall-clock passage beyond bucket_end alone is never closure evidence. For a
historical read, immutable valid source partitions, their declared coverage,
content/manifest identities and strict gap-free selection can provide the
required finalization evidence. For incremental/live operation, the source or
collector must provide a named completeness/watermark/finalization signal.
The exact live collector protocol is out of scope, but it may not be omitted
from a future implementation.
10. Late events, corrections and revisions¶
An event whose exchange_ts is inside a bucket but arrives after the bucket's
wall-clock end is still accepted into the mutable PARTIAL state if the
bucket has not received closure evidence. Apparent boundary passage does not
seal it.
After a bucket is CLOSED, a newly discovered event, source correction or changed source partition revision must never silently mutate that CLOSED record in place. The required behavior is:
- retain the old CLOSED result as an immutable result of its old source snapshot/revision;
- publish or select a new source revision/content identity;
- rebuild every affected bucket from the complete revised source support;
- produce a new CLOSED result/materialization revision with lineage to the revised source and, where applicable, the superseded result.
An implementation may mark the old result superseded in a catalog/application view, but supersession is not mutation and the old bytes/meaning remain reproducible. Source partition revisions and materialization revisions are provenance/result concepts; neither is added to CandleDefinition identity.
The candle runtime does not define live source precedence, deduplication or repair orchestration. It must, however, honor the immutable-closed and revisioned-rebuild obligations above.
11. Temporal availability and lookahead prevention¶
The relevant times are distinct:
| Time | Meaning |
|---|---|
trade exchange_ts |
venue event time used for bucket membership |
bucket_start / bucket_end |
representation interval boundaries |
record_time |
canonical candle coordinate, equal to bucket_start |
causal_floor |
deterministic minimum logical time before which the final CLOSED result may not be consumed; v1 is bucket_end |
observed_available_at |
actual observed source/finalization availability time when evidence exists; nullable/unknown otherwise |
For every CLOSED candle, causal_floor is exactly bucket_end. If
observed_available_at is present, it must be no earlier than that floor and
must represent actual observed source/finalization availability. A
Feature/Research/Strategy consumer must not consume final CLOSED values before
the applicable causal floor or observed availability constraint. CLOSED
does not imply observed_available_at == bucket_end.
Historical trade-v1 imports generally have no real receive_ts. They
therefore cannot establish the wall-clock time at which a live observer could
have known the result. observed_available_at is consequently nullable/unknown
unless adequate observed source/finalization evidence exists; the contract
never manufactures it from exchange time, file/import time or row order. For
historical causal replay, bucket_end is the deterministic logical causal
floor, not a claim that the data was actually available then. A future replay
may explicitly adopt an exchange-time-only availability assumption, but that
assumption must not be presented as observed live availability or
latency-faithful replay. When actual source finalization/observation evidence
exists, it must remain in provenance and the later real time governs.
PARTIAL has no final-candle availability and is not a closed-candle input.
bucket_start must never be used as a proxy for final value availability.
12. Historical/live equivalence¶
The core convergence invariant is:
same CandleDefinition
+ semantically equivalent complete source events
+ same source revision/content
= same final CLOSED candle records
This equality holds regardless of whether construction is batch historical, incremental, on demand or sourced from an approved materialized closed series. Intermediate PARTIAL states and update timing need not be byte-identical. Final closed records, bucket membership, exact decimals, ordering-dependent open/ close, empty-bucket omission and coverage meaning must be equivalent.
13. Materialization equivalence¶
An on-demand CLOSED result and a catalog-backed materialized result are semantically equivalent when they have:
- the same CandleDefinition identity;
- the same candidate bucket set and returned query semantics;
- the same source snapshot/revision/content evidence;
- the same required bucket support and coverage result;
- the same canonical
candle-v1records, including omission of empty buckets; - compatible availability/finalization evidence.
Materialization is a derived dataset operation, not a new candle meaning. Physical storage root, relative path, file format, compression, partition placement, catalog UUID and process/cache identity are not semantic identity. Materialized output must retain source lineage and provenance. An implementation identity is recorded to explain how computation occurred, but does not make otherwise equal semantic records unequal; differing output is a contract failure exposed by content/result identity, not a new definition.
14. Provenance minimum¶
Every reusable CLOSED result, and every on-demand result that claims reproducibility, must retain at least:
CandleDefinition identity and version
source representation identity: trades@1 / trade-v1
resolved natural source DatasetIdentity
ordered NaturalPartitionIdentity values and revisions
source manifest/content hashes or equivalent immutable content evidence
source result/snapshot identity when one exists
requested interval and required bucket-support interval(s)
source coverage and finalization evidence
implementation/code identity used for computation
materialization/result identity when persisted
Catalog UUIDs, storage roots, relative paths and absolute paths may be
diagnostic runtime metadata but cannot enter semantic identity or be the only
provenance needed to reproduce a result. Missing receive_ts remains missing.
15. Canonical output record¶
The proposed schemas/candle-v1.json defines one non-empty CLOSED record:
| Field | Meaning |
|---|---|
bucket_start |
UTC canonical start of the half-open bucket; record time |
bucket_end |
UTC canonical end; exactly bucket_start + duration_ns |
open |
first source-ordered trade price |
high |
exact-decimal maximum trade price |
low |
exact-decimal minimum trade price |
close |
last source-ordered trade price |
volume |
exact-decimal sum of trade sizes |
trade_count |
exact count of source trade records, canonical digit string |
All fields are required and no additional fields are accepted by the proposed
record schema. state, availability, source provenance and definition
identity belong in the runtime/result envelope, not in each row. A partial
runtime object is not a candle-v1 materialized record.
16. Explicit non-goals and evolution¶
This contract does not define:
- an aggregator class, package, execution schedule or live collector;
- calendar/month/session alignment;
- volume/tick/dollar bars;
- footprint, book or L1/L2/L3 representations;
- source deduplication, precedence, repair or correction orchestration;
- DataGateway, catalog DDL, producer coverage population or storage layout;
- API transport, UI/TUI/CLI behavior or feature/indicator semantics.
Any semantic change to v1, including changing empty buckets, timestamp coordinate, source order, numerical rounding, query selection, closure or availability, requires an explicit new version under ADR-0018. The runtime mandate must implement this contract as written after independent review; it must not use implementation defaults to fill gaps in the contract.