ADR-0043 — Live-ingest server v1 composition¶
Status: ACCEPTED Date: 2026-09-25
Context¶
SCOPE.md ("Live Ingest Server Production Readiness v1") Active Path step 2
requires a design gate that closes finding P1 — Persistent Ingest Server
Ownership before any implementation slice may build a bounded, long-running
live-ingest process (issue #124, blocking issue #126).
A11 (ADR-0040), K02 (ADR-0041) and K10 (ADR-0042) are frozen and implemented,
and their real-server restart/reconcile/checkpoint-advance path is proven by
PR #122. But every existing entrypoint that exercises them --
tools/bybit_live_proof.py, tools/live_real_publish_proof.py,
tools/live_real_restart_proof.py -- is a one-shot, bounded CLI invocation
that runs to completion and exits. There is no persistent-loop, daemon or
service scaffolding anywhere in the repository. PR #122's own "Process A" /
"Process B" real-server evidence is realized purely operationally: the same
restart-proof script is launched twice as two independent OS processes,
coupled only through the durable on-disk checkpoint file and durable catalog
state -- there is no in-process loop and no IPC.
docs/architecture/MARKET_DATA_INGEST.md already anticipated this gap ("A
live collector is not required to use the same implementation as a finite
job... The deployment topology remains open... does not prescribe a
container count, process topology, broker, scheduler, async framework, or
checkpoint database") and left it as an open decision. This ADR closes that
decision for the first bounded vertical, without freezing anything beyond it.
K03/K04/K05 observability, capacity and pressure primitives
(quant_platform.operations.observability, .capacity, .pressure) are
already implemented and credited COMPLETE, but are not wired into
bybit_live.py or operations/checkpoint.py anywhere -- today's only
operator-visible signal is print() output from the one-shot proof CLIs.
Decision¶
1. Composition: reuse the existing restart/reconcile cycle as the loop's steady state¶
The key structural insight is that PR #122's "Process A publishes; Process B restarts, reconciles and advances the checkpoint" sequence is not just proof scaffolding -- it is structurally identical to two consecutive iterations of a continuous server. The server loop is defined as:
on start:
load checkpoint (if any)
resume_live_ingest(...) # exact ADR-0042 S5 restart procedure,
# reused unchanged (existing function)
-> RestartOutcome: NO_CHECKPOINT | GAP_DETECTED | RESUMED
repeat until stop requested:
acquire one bounded batch # existing acquisition seam
canonicalize/publish/certify/catalog # existing S13/S14 composition
advance checkpoint # only after durable publication
emit K03 operational signal(s)
if the session dropped:
loop back to resume_live_ingest # same reconcile path as `on start`,
# not a new disconnect model
No new acquisition, canonicalization, publication, certification, cataloging
or checkpoint-binding logic is introduced. The loop is a thin, unbounded
outer while composing the exact functions already implemented in
quant_platform.application.bybit_live:
- startup/gap recovery:
resume_live_ingest,load_current_durable_publication_state; - each cycle's acquire+publish: the same composition
run_real_server_publish_proofalready performs (bounded acquisition -> canonical materialization -> S13 certification -> S14 publication/catalog -> eligibility), reused with production-sized bounds instead of proof-sized bounds (max_messages/max_secondsbecome operator-configured, not test constants); - checkpoint advance:
next_checkpoint/_persist_checkpoint_from_publication, called only when the preceding publish reached a durable/eligible status.
A disconnect between cycles is not a new failure mode to design: it is exactly the disconnect/reconnect case ADR-0040 §7 and ADR-0042 §5 already freeze, and it is exercised the same way whether the previous process exited (as in the two-OS-process proof) or the same process is simply starting its next cycle.
2. Owning module and executable boundary¶
Per the existing, already-frozen DG-D/C05 rule (OPEN_DECISIONS.md:
"executable tooling owns CLI/environment acquisition and parsing...
quant_platform.application owns concrete composition and does not read
process arguments/environment directly"), the loop follows the same split
already used by every other live-ingest entrypoint:
quant_platform.application.live_ingest_server(new module) owns the loop composition described in §1: a typed, immutableLiveIngestServerConfigV1(storage root/id, dsn, checkpoint path, producer/code-ref, per-cycle acquisition bounds, stop-check interval) and arun_live_ingest_server(config, *, stop_requested)entrypoint, wherestop_requestedis a caller-supplied zero-argument predicate (defaults to "never"), so the loop itself never touchessignal/OS process control -- that stays at the executable boundary, consistent with C05.tools/live_ingest_server.py(new script) owns argv/env parsing and OS signal wiring only: it buildsLiveIngestServerConfigV1from CLI arguments (mirroringtools/live_real_restart_proof.py's existing flags:--dsn,--storage-root,--storage-root-id,--checkpoint-path,--producer,--code-ref, plus new--max-messages-per-cycle,--max-seconds-per-cycle), installsSIGTERM/SIGINThandlers that flip a stop flag (never a hard kill of an in-flight publish), prints the same_identity_evidence()blocklive_real_restart_proof.pyalready prints, and callsrun_live_ingest_server.
No __main__.py, console_scripts entry point or process manager is added
by this ADR; how the operator supervises/restarts the tools/live_ingest_server.py
process (systemd unit, supervisord, container restart policy, or a plain
foreground process during first rollout) is deployment-local, exactly as
ADR-0041 §1/§4 already treats "exact OS/service-manager primitives."
3. Publication-before-checkpoint stays enforced by existing code, not new loop logic¶
The loop must never call checkpoint-advance except immediately after a
publish call returns a durable/eligible status, using the same
_persist_checkpoint_from_publication / next_checkpoint functions that
already enforce ADR-0042 §2's ordering and §4's monotonicity/domain rules. A
clean stop between cycles (stop flag observed after a cycle completes) is
already covered by ADR-0042 §3's "crash after checkpoint advance" case; the
loop introduces no new crash-state.
4. Operator/runbook evidence (closes P6 for this design gate)¶
The loop must wire the already-implemented, currently-unused K03 seam
(quant_platform.operations.observability) rather than continue with bare
print():
- emit one
OperationalSignalV1withSignalKind.LIFECYCLE_TRANSITIONon every loop-state change (STARTING -> RESUMING -> RUNNING -> STOPPING -> STOPPED, plus entry into a reconcile/gap-recovery cycle); - emit one
OperationalSignalV1withSignalKind.HEALTH_SNAPSHOTper completed cycle, mapping existing return values toHealthState:SessionState.ACQUIRING/durable publish PASS ->HEALTHY; explicit non-complete gap (RestartOutcome.status == GAP_DETECTED) ->DEGRADED; publish/certification failure ->FAILED; a cycle that could not observe its own outcome ->UNKNOWN(never silentlyHEALTHY, per the seam's own frozen rule); - emit
SignalKind.FAILUREon any unhandled cycle exception before the process exits; - reuse
observe_capacity/evaluate_pressure(K04/K05, already implemented) once per cycle and fold theirPressureDecisioninto the sameHEALTH_SNAPSHOTsignal'sdiagnostics, rather than inventing a parallel pressure check.
At minimum, the resulting signal stream must let an operator read: current
session/subscription state; last durable checkpoint identity and generation;
last durable canonical TradeKeyV1; publication/certification/catalog
failures; reconnect/reconciliation outcome; explicit gap intervals; pressure
and capacity decisions; and confirmation the process is running under the
K02 mkt-transform identity (the startup _identity_evidence() print,
reused unchanged).
Where the signal stream is sent (stdout, file, log aggregator) is deployment-local and not decided here, matching P6's own text ("exact logging, service manager and alerting technology remain deployment-local unless a future ADR freezes them"); only the in-repo emission call sites are decided by this ADR.
5. Why this does not require a generic scheduler, broker, workflow engine or J03¶
The loop is a single bounded process for exactly one venue, instrument and
schema (bybit/linear/BTCUSDT/trade-v1), with no submission identity,
no independent job queue or result store, no retry/backoff policy beyond the
reconnect/reconciliation semantics ADR-0040 already freezes, and no
multi-tenant or multi-job scheduling concern. If the process itself dies, the
deployment-local service manager restarts it, and the existing checkpoint
resumes it -- this is process supervision, not job orchestration. J03 (job
runtime: submission identity, retry/idempotency, result identity, failure
semantics for arbitrary durable operations) is a different, general-purpose
capability this ADR does not need and does not touch.
6. Why this does not imply J08 live product mode¶
J08 requires J07 (paper/shadow mode) and K09 (retention/deletion authority) in addition to K02/K03/K05/K06/K08/K10, none of which this ADR touches or requires. The server has no order-placement, execution, portfolio or trading authority; it only continuously performs what A11/K10 already do as bounded one-shot proofs today. Reaching this ADR's design does not authorize or imply any J08 proposition.
7. K02 conformance¶
The process runs under the same least-privilege mkt-transform identity
already proven for the one-shot proofs (PR #113): the same checkpoint-path
filesystem authority, the same minimum catalog/database role, and no new
provider secret (Bybit public trades v1 needs none, ADR-0041 §2). Running
continuously does not widen filesystem, database, backup or administrative
authority beyond what ADR-0041 already freezes.
Excluded¶
This ADR does not define or authorize:
- a generic scheduler, broker, workflow engine or J03 job runtime;
- J08 live product mode or any execution/trading authority;
- B06 live-consumer/DataGateway cursor semantics;
- K07 tier relocation or K09 retention/deletion authority;
- second venue, second instrument or generic provider resolution;
- any new canonicalization, publication, certification, cataloging or checkpoint-binding semantics beyond ADR-0040/ADR-0041/ADR-0042;
- long-gap repair/state-machine design (Active Path step 3, ADR to follow separately per issue #125);
- the exact OS/service-manager unit, container image or log sink used to run
tools/live_ingest_server.pyin production (deployment-local).
Consequences¶
- Active Path step 4 (issue #126) can implement
LiveIngestServerConfigV1,run_live_ingest_serverandtools/live_ingest_server.pyby composing the functions named in §1 into the loop shape defined here, without further semantic decisions about acquisition, publication, checkpoint ordering or observability wiring. - Hermetic tests can exercise multiple loop iterations with a fake/bounded
stop predicate and the existing checkpoint/publish test doubles, verifying
cycle ordering (resume -> acquire -> publish -> checkpoint -> signal) and
clean-stop behavior, the same way
tests/test_bybit_live_checkpoint_v1.pyalready exercises the two-phase restart composition. - Real-server continuous-run, stop/restart and signal-emission evidence remains the responsibility of Active Path step 6 (issue #128), not this design gate.
docs/architecture/MARKET_DATA_INGEST.md§12/§14's "deployment topology remains open" / "runtime checkpoint and deployment choices" open items are now closed for this first vertical by this ADR; the document's broader statement that a future second collector need not share this implementation is unaffected.