LIPR-05 - Live Ingest Server Production-Readiness Proof¶
- Status: PASS for target bounded run/restart/reconcile/readback; long-gap forced path covered by equivalent hermetic simulation.
- Issue: #128
- Scope:
SCOPE.md/ Live Ingest Server Production Readiness v1 / Active Path step 6. - Branch:
codex/issue-128-lipr05-production-proof - Evidence date: 2026-09-25
This document records the proof work for the bounded Bybit BTCUSDT live-ingest
server. Target-server evidence was executed by the operator on homelab under
the K02 mkt-transform runtime identity after Codex prepared the proof
controls and command pack.
Authority¶
The proof is bounded to:
- ADR-0040 - Bybit live trades acquisition v1;
- ADR-0041 - live-ingest runtime identity v1;
- ADR-0042 - live-ingest checkpoint/recovery v1;
- ADR-0043 - live-ingest server v1 composition;
- ADR-0044 - live-ingest long-gap remediation and explicit-gap state v1;
- SCOPE acceptance items 1-11 for Live Ingest Server Production Readiness v1.
Explicit exclusions remain unchanged: B06 live-consumer cursor, K07 tier relocation, K09 deletion authority, J08 live product mode, strategy/execution, paper/shadow operation, second venues and generic schedulers.
Implemented Proof Control¶
tools/live_ingest_server.py now accepts:
Default behavior is unchanged. When supplied, the executable boundary requests
a clean stop after N completed HEALTH_SNAPSHOT cycles. This keeps bounded
production-readiness proofs deterministic without hard-killing an in-flight
publish and without adding acquisition, publication, checkpoint or repair
semantics outside the existing Application boundary.
Executed Local Evidence¶
CLI proof-control and server/gap tests¶
Command:
python -m unittest tests.test_live_ingest_server_cli_v1 tests.test_live_ingest_server_v1 tests.test_live_gap_orchestration_v1
Observed output:
Coverage from these tests:
- clean bounded stop through the CLI
--max-cyclesproof control; - steady-state live acquisition cycles use the live publish path, not REST polling;
- startup with an existing checkpoint reconciles before steady-state acquisition;
- bounded reconcile resumes when continuity is provable;
- unresolved long gap records durable
known_gapcoverage with no partitions; - long-gap state remains visible after subsequent restart/checkpoint advance;
- repeated sustained gap updates one open supersession chain;
- unparseable durable long-gap manifests emit
OBSERVATION_UNAVAILABLE; - corrupt checkpoint fails closed through a
FAILUREsignal.
Package-boundary verification¶
Command:
Observed output:
This confirms the new proof-control test and CLI change preserve the existing
executable-boundary rule: tools/ reaches repository runtime code through
quant_platform.application.
CLI help surface¶
Command:
Material observed line:
--max-cycles MAX_CYCLES
Stop cleanly after this many completed health-snapshot cycles; intended for bounded proofs
Target-Server Preflight¶
Command:
ssh -o BatchMode=yes -o ConnectTimeout=8 neonix@homelab.local \
"cd /opt/market-platform && git fetch origin governance/ingest-server-production-v1-scope && git rev-parse --short FETCH_HEAD"
Observed output:
Command:
Observed output:
Result: Codex can reach the target server and fetch Git authority, but cannot
execute the K02-valid runtime identity from this non-interactive session.
Running as neonix would not satisfy ADR-0041/K02 because the proof must run
under the least-privileged mkt-transform identity.
The operator then executed the target proof locally with interactive sudo authorization. Runtime evidence below is copied from that execution.
Target Proof Execution¶
Run:
RUN_ID=lipr05-live-ingest-server-20260925T192648Z
WORKTREE=/tmp/lipr05-live-ingest-server-20260925T192648Z-worktree
CHECKPOINT=/srv/marketdata/canonical/_k10_checkpoints/lipr05-live-ingest-server-20260925T192648Z.json
EVIDENCE_DIR=/tmp/lipr05-live-ingest-server-20260925T192648Z-evidence
branch head=699b0ed
The command pack used for the proof kept secrets out of Git and wrote only bounded logs/JSON metadata, not bulk market data.
cd /opt/market-platform
sudo -v
set -euo pipefail
git fetch origin codex/issue-128-lipr05-production-proof
RUN_ID="lipr05-live-ingest-server-$(date -u +%Y%m%dT%H%M%SZ)"
WORKTREE="/tmp/${RUN_ID}-worktree"
CHECKPOINT="/srv/marketdata/canonical/_k10_checkpoints/${RUN_ID}.json"
EVIDENCE_DIR="/tmp/${RUN_ID}-evidence"
PGPASS="/tmp/${RUN_ID}.pgpass"
git worktree add --detach "$WORKTREE" FETCH_HEAD
mkdir -p "$EVIDENCE_DIR"
read -rsp "market_catalog_writer password: " K10_DB_PASSWORD
echo
sudo -u mkt-transform sh -c "umask 077; cat > '$PGPASS'" <<EOF
127.0.0.1:5433:market_catalog:market_catalog_writer:${K10_DB_PASSWORD}
EOF
unset K10_DB_PASSWORD
date -Is | tee "$EVIDENCE_DIR/process-a-start.txt"
sudo -u mkt-transform env \
PGPASSFILE="$PGPASS" \
PGHOST=127.0.0.1 \
PGPORT=5433 \
PGUSER=market_catalog_writer \
PGDATABASE=market_catalog \
/opt/market-platform/.venv/bin/python "$WORKTREE/tools/live_ingest_server.py" \
--storage-root /srv/marketdata \
--storage-root-id hot \
--checkpoint-path "$CHECKPOINT" \
--producer "lipr05-live-ingest-server" \
--code-ref "issue-128-${RUN_ID}" \
--max-messages-per-cycle 1 \
--max-seconds-per-cycle 15 \
--cycle-interval-seconds 0 \
--max-cycles 1 \
> "$EVIDENCE_DIR/process-a.stdout" 2>&1
date -Is | tee "$EVIDENCE_DIR/process-a-end.txt"
date -Is | tee "$EVIDENCE_DIR/process-b-start.txt"
sudo -u mkt-transform env \
PGPASSFILE="$PGPASS" \
PGHOST=127.0.0.1 \
PGPORT=5433 \
PGUSER=market_catalog_writer \
PGDATABASE=market_catalog \
/opt/market-platform/.venv/bin/python "$WORKTREE/tools/live_ingest_server.py" \
--storage-root /srv/marketdata \
--storage-root-id hot \
--checkpoint-path "$CHECKPOINT" \
--producer "lipr05-live-ingest-server" \
--code-ref "issue-128-${RUN_ID}" \
--max-messages-per-cycle 1 \
--max-seconds-per-cycle 15 \
--cycle-interval-seconds 0 \
--max-cycles 1 \
> "$EVIDENCE_DIR/process-b.stdout" 2>&1
date -Is | tee "$EVIDENCE_DIR/process-b-end.txt"
sudo -u mkt-transform ls -l "$CHECKPOINT" | tee "$EVIDENCE_DIR/checkpoint-ls.txt"
sudo -u mkt-transform cat "$CHECKPOINT" | tee "$EVIDENCE_DIR/checkpoint-after-b.json"
grep -E "effective runtime identity|user=mkt-transform|is_root=False|signal kind=|LIVE_INGEST_SERVER: STOPPED|max cycles reached" \
"$EVIDENCE_DIR/process-a.stdout" "$EVIDENCE_DIR/process-b.stdout" \
| tee "$EVIDENCE_DIR/material-summary.txt"
rm -f "$PGPASS"
Expected acceptance signals from the command pack:
- both processes print
user=mkt-transformandis_root=False; - Process A reaches
LIVE_INGEST_SERVER: STOPPEDafter a bounded acquire cycle and persists a checkpoint; - Process B starts from the same checkpoint, emits a reconcile health snapshot before any new steady-state acquisition, and exits cleanly;
- checkpoint generation and identity after Process B prove publication before checkpoint advance;
- pressure/capacity fields appear in
HEALTH_SNAPSHOTpayloads; - no
known_gapinterval is marked complete unless a future A10 repair cutover evidence path proves it.
Process A - bounded acquire¶
Material output:
RC_A=0
user=mkt-transform
primary_group=marketdata
is_root=False
signal kind=HEALTH_SNAPSHOT ... checkpoint_identity=live-checkpoint-v1:sha256:dd0a2181d1c58c5c893ce072b01f841919f9bb6481ecb792db2394ce65dc0ce7,cycle=1,health=HEALTHY,phase=acquire,pressure_new_writes_allowed=True,pressure_state=NORMAL,session_state=CONTINUOUS
LIVE_INGEST_SERVER: STOPPED
Acceptance covered:
- K02-compatible runtime identity (
mkt-transform, non-root); - normal bounded acquire cycle reached
HEALTHY; - checkpoint advanced only after the cycle's durable publication path;
- K04/K05 capacity/pressure evidence appeared in the health signal.
Process B - restart/reconcile¶
Material output:
RC_B=0
user=mkt-transform
primary_group=marketdata
is_root=False
signal kind=HEALTH_SNAPSHOT ... checkpoint_identity=live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d,cycle=1,health=HEALTHY,phase=reconcile,pressure_new_writes_allowed=True,pressure_state=NORMAL,records=769,session_state=CONTINUOUS
LIVE_INGEST_SERVER: STOPPED
Acceptance covered:
- restart used the existing checkpoint and entered
phase=reconcile; - bounded reconciliation accepted 769 records and returned
HEALTHY; - session stayed
CONTINUOUS; - K04/K05 capacity/pressure evidence remained
NORMAL; - stop was clean after the bounded proof cycle.
Final checkpoint¶
Material output:
{
"checkpoint_identity": "live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d",
"coverage_segment_id": "k10-restart-reconciliation-20260925-193111",
"created_at": "2026-09-25T19:31:12.051818Z",
"dataset_identity": {
"dataset_kind": "trades",
"instrument": "BTCUSDT",
"layer": "canonical",
"record_schema_id": "trade-v1",
"venue": "bybit"
},
"generation": 2,
"last_canonical_exchange_ts": "2026-09-25T19:31:08.815Z",
"last_canonical_trade_id": "e7ea1e1d-bf73-581f-bf03-67adba52c9f0",
"last_observed_sequence": "816124801187",
"partition_key": "dt=2026-09-25",
"revision": 2,
"source_semantics_id": "bybit-public-trades-websocket-v1"
}
Acceptance covered:
- checkpoint generation advanced from Process A to Process B (
generation=2); - the final checkpoint binds the Bybit BTCUSDT canonical
trade-v1dataset; - the checkpoint binds the restart reconciliation segment and revision 2.
DataGateway readback¶
Command:
sudo -u mkt-transform env PGPASSFILE="$PGPASS" \
CATALOG_DSN="host=127.0.0.1 port=5433 dbname=market_catalog user=market_catalog_writer" \
CHECKPOINT="$CHECKPOINT" \
/opt/market-platform/.venv/bin/python /tmp/lipr05-readback-check.py \
| tee "$EVIDENCE_DIR/datagateway-readback-check.stdout"
Observed output:
{
"catalog_partition_ids": [
"189e296c-c71a-40dc-9b57-20f1a08303c2"
],
"checkpoint_generation": 2,
"checkpoint_identity": "live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d",
"checkpoint_last_trade": {
"exchange_ts": "2026-09-25T19:31:08.815Z",
"partition_key": "dt=2026-09-25",
"sequence": "816124801187",
"trade_id": "e7ea1e1d-bf73-581f-bf03-67adba52c9f0"
},
"checkpoint_revision": 2,
"contains_checkpoint_anchor": true,
"coverage_complete": false,
"coverage_gaps": [
{
"end": "2026-09-25T19:31:08.816Z",
"start": "2026-09-25T19:31:08.815001Z"
}
],
"duplicate_trade_key_count": 0,
"rel_paths": [
"dt=2026-09-25/part-193111.parquet"
],
"request_end": "2026-09-25T19:31:08.816Z",
"request_start": "2026-09-25T19:31:08.815Z",
"rows_read": 4,
"status": "PASS",
"storage_root_ids": [
"hot"
]
}
Acceptance covered:
- DataGateway readback found the checkpoint anchor;
- no duplicate canonical economic
TradeKeyV1was found in the readback slice; - metadata did not falsely claim complete coverage for an interval with an
explicit uncovered tail (
coverage_complete=false,coverage_gapspresent); - readback stayed on the
hotstorage root and revision 2 restart partition.
Forced Long-Gap Evidence¶
Real forced long-gap execution was NOT EXECUTED on the target server. Issue #128 allows a forced long-gap scenario or an explicitly accepted equivalent simulation; the repository proof uses the equivalent hermetic simulation below so the target server is not intentionally degraded.
Equivalent hermetic coverage is currently:
tests.test_live_ingest_server_v1.test_gap_detected_on_startup_records_non_complete_gap_and_starts_new_segment;tests.test_live_ingest_server_v1.test_open_gap_observability_survives_restart_after_checkpoint_advances;tests.test_live_ingest_server_v1.test_sustained_gap_updates_one_open_gap_chain_instead_of_appending_events;tests.test_live_gap_orchestration_v1.test_records_known_gap_without_partition_and_unproven_source_evaluation;tests.test_live_gap_orchestration_v1.test_inconclusive_source_evaluation_is_retryable_and_not_unproven.
These tests prove the accepted equivalent simulation path: unresolved restart
continuity records explicit non-complete known_gap coverage, preserves open
gap observability across restart/checkpoint advance, supersedes repeated
detections in one open chain, and keeps inconclusive source evaluation
distinct from a completed negative result.
Acceptance State¶
| Requirement | State | Evidence |
|---|---|---|
| Server runs under K02-compatible authority | PASS | Process A/B show user=mkt-transform, is_root=False |
| Publication-before-checkpoint during normal running | PASS | Process A health checkpoint plus final checkpoint generation |
| Publication-before-checkpoint during restart/reconcile | PASS | Process B phase=reconcile, final generation 2/revision 2 |
| Bounded reconcile resumes when continuity is provable | PASS | Process B records=769, health=HEALTHY, session_state=CONTINUOUS |
| Long-gap remains explicit/non-complete | PASS for equivalent simulation | Hermetic tests listed above |
| No silent loss / no duplicate canonical economic trades | PASS | DataGateway readback contains_checkpoint_anchor=true, duplicate_trade_key_count=0 |
| No false complete coverage | PASS | DataGateway readback coverage_complete=false with explicit coverage_gaps |
| Operator/runbook evidence covers session/publication/checkpoint/gap/repair/pressure/authority | PASS | Process logs, final checkpoint, readback JSON, hermetic gap tests |
| No secrets/bulk market data committed | PASS | This document records commands and bounded outputs only |
Current Conclusion¶
Issue #128's production-readiness proof is satisfied for the bounded live-ingest server v1 path, assuming the documented hermetic long-gap tests are accepted as the issue's allowed equivalent simulation. The target server proved K02 runtime identity, bounded acquire, stop/restart, bounded reconcile, checkpoint generation advance, DataGateway readback of the checkpoint anchor, zero duplicate canonical economic trade keys in the readback slice, and no false complete coverage claim.