Skip to content

LIPR-05 - Live Ingest Server Production-Readiness Proof

  • Status: PASS for target bounded run/restart/reconcile/readback; long-gap forced path covered by equivalent hermetic simulation.
  • Issue: #128
  • Scope: SCOPE.md / Live Ingest Server Production Readiness v1 / Active Path step 6.
  • Branch: codex/issue-128-lipr05-production-proof
  • Evidence date: 2026-09-25

This document records the proof work for the bounded Bybit BTCUSDT live-ingest server. Target-server evidence was executed by the operator on homelab under the K02 mkt-transform runtime identity after Codex prepared the proof controls and command pack.

Authority

The proof is bounded to:

  • ADR-0040 - Bybit live trades acquisition v1;
  • ADR-0041 - live-ingest runtime identity v1;
  • ADR-0042 - live-ingest checkpoint/recovery v1;
  • ADR-0043 - live-ingest server v1 composition;
  • ADR-0044 - live-ingest long-gap remediation and explicit-gap state v1;
  • SCOPE acceptance items 1-11 for Live Ingest Server Production Readiness v1.

Explicit exclusions remain unchanged: B06 live-consumer cursor, K07 tier relocation, K09 deletion authority, J08 live product mode, strategy/execution, paper/shadow operation, second venues and generic schedulers.

Implemented Proof Control

tools/live_ingest_server.py now accepts:

--max-cycles N

Default behavior is unchanged. When supplied, the executable boundary requests a clean stop after N completed HEALTH_SNAPSHOT cycles. This keeps bounded production-readiness proofs deterministic without hard-killing an in-flight publish and without adding acquisition, publication, checkpoint or repair semantics outside the existing Application boundary.

Executed Local Evidence

CLI proof-control and server/gap tests

Command:

python -m unittest tests.test_live_ingest_server_cli_v1 tests.test_live_ingest_server_v1 tests.test_live_gap_orchestration_v1

Observed output:

Ran 20 tests in 0.210s
OK

Coverage from these tests:

  • clean bounded stop through the CLI --max-cycles proof control;
  • steady-state live acquisition cycles use the live publish path, not REST polling;
  • startup with an existing checkpoint reconciles before steady-state acquisition;
  • bounded reconcile resumes when continuity is provable;
  • unresolved long gap records durable known_gap coverage with no partitions;
  • long-gap state remains visible after subsequent restart/checkpoint advance;
  • repeated sustained gap updates one open supersession chain;
  • unparseable durable long-gap manifests emit OBSERVATION_UNAVAILABLE;
  • corrupt checkpoint fails closed through a FAILURE signal.

Package-boundary verification

Command:

python -m unittest tests.test_package_boundaries_v1

Observed output:

Ran 15 tests in 9.064s
OK

This confirms the new proof-control test and CLI change preserve the existing executable-boundary rule: tools/ reaches repository runtime code through quant_platform.application.

CLI help surface

Command:

python tools/live_ingest_server.py --help

Material observed line:

--max-cycles MAX_CYCLES
  Stop cleanly after this many completed health-snapshot cycles; intended for bounded proofs

Target-Server Preflight

Command:

ssh -o BatchMode=yes -o ConnectTimeout=8 neonix@homelab.local \
  "cd /opt/market-platform && git fetch origin governance/ingest-server-production-v1-scope && git rev-parse --short FETCH_HEAD"

Observed output:

bf4c5d6

Command:

ssh -o BatchMode=yes -o ConnectTimeout=8 neonix@homelab.local \
  "sudo -n -u mkt-transform id"

Observed output:

sudo: password is required

Result: Codex can reach the target server and fetch Git authority, but cannot execute the K02-valid runtime identity from this non-interactive session. Running as neonix would not satisfy ADR-0041/K02 because the proof must run under the least-privileged mkt-transform identity.

The operator then executed the target proof locally with interactive sudo authorization. Runtime evidence below is copied from that execution.

Target Proof Execution

Run:

RUN_ID=lipr05-live-ingest-server-20260925T192648Z
WORKTREE=/tmp/lipr05-live-ingest-server-20260925T192648Z-worktree
CHECKPOINT=/srv/marketdata/canonical/_k10_checkpoints/lipr05-live-ingest-server-20260925T192648Z.json
EVIDENCE_DIR=/tmp/lipr05-live-ingest-server-20260925T192648Z-evidence
branch head=699b0ed

The command pack used for the proof kept secrets out of Git and wrote only bounded logs/JSON metadata, not bulk market data.

cd /opt/market-platform
sudo -v
set -euo pipefail

git fetch origin codex/issue-128-lipr05-production-proof

RUN_ID="lipr05-live-ingest-server-$(date -u +%Y%m%dT%H%M%SZ)"
WORKTREE="/tmp/${RUN_ID}-worktree"
CHECKPOINT="/srv/marketdata/canonical/_k10_checkpoints/${RUN_ID}.json"
EVIDENCE_DIR="/tmp/${RUN_ID}-evidence"
PGPASS="/tmp/${RUN_ID}.pgpass"

git worktree add --detach "$WORKTREE" FETCH_HEAD
mkdir -p "$EVIDENCE_DIR"

read -rsp "market_catalog_writer password: " K10_DB_PASSWORD
echo
sudo -u mkt-transform sh -c "umask 077; cat > '$PGPASS'" <<EOF
127.0.0.1:5433:market_catalog:market_catalog_writer:${K10_DB_PASSWORD}
EOF
unset K10_DB_PASSWORD

date -Is | tee "$EVIDENCE_DIR/process-a-start.txt"
sudo -u mkt-transform env \
  PGPASSFILE="$PGPASS" \
  PGHOST=127.0.0.1 \
  PGPORT=5433 \
  PGUSER=market_catalog_writer \
  PGDATABASE=market_catalog \
  /opt/market-platform/.venv/bin/python "$WORKTREE/tools/live_ingest_server.py" \
    --storage-root /srv/marketdata \
    --storage-root-id hot \
    --checkpoint-path "$CHECKPOINT" \
    --producer "lipr05-live-ingest-server" \
    --code-ref "issue-128-${RUN_ID}" \
    --max-messages-per-cycle 1 \
    --max-seconds-per-cycle 15 \
    --cycle-interval-seconds 0 \
    --max-cycles 1 \
  > "$EVIDENCE_DIR/process-a.stdout" 2>&1
date -Is | tee "$EVIDENCE_DIR/process-a-end.txt"

date -Is | tee "$EVIDENCE_DIR/process-b-start.txt"
sudo -u mkt-transform env \
  PGPASSFILE="$PGPASS" \
  PGHOST=127.0.0.1 \
  PGPORT=5433 \
  PGUSER=market_catalog_writer \
  PGDATABASE=market_catalog \
  /opt/market-platform/.venv/bin/python "$WORKTREE/tools/live_ingest_server.py" \
    --storage-root /srv/marketdata \
    --storage-root-id hot \
    --checkpoint-path "$CHECKPOINT" \
    --producer "lipr05-live-ingest-server" \
    --code-ref "issue-128-${RUN_ID}" \
    --max-messages-per-cycle 1 \
    --max-seconds-per-cycle 15 \
    --cycle-interval-seconds 0 \
    --max-cycles 1 \
  > "$EVIDENCE_DIR/process-b.stdout" 2>&1
date -Is | tee "$EVIDENCE_DIR/process-b-end.txt"

sudo -u mkt-transform ls -l "$CHECKPOINT" | tee "$EVIDENCE_DIR/checkpoint-ls.txt"
sudo -u mkt-transform cat "$CHECKPOINT" | tee "$EVIDENCE_DIR/checkpoint-after-b.json"

grep -E "effective runtime identity|user=mkt-transform|is_root=False|signal kind=|LIVE_INGEST_SERVER: STOPPED|max cycles reached" \
  "$EVIDENCE_DIR/process-a.stdout" "$EVIDENCE_DIR/process-b.stdout" \
  | tee "$EVIDENCE_DIR/material-summary.txt"

rm -f "$PGPASS"

Expected acceptance signals from the command pack:

  • both processes print user=mkt-transform and is_root=False;
  • Process A reaches LIVE_INGEST_SERVER: STOPPED after a bounded acquire cycle and persists a checkpoint;
  • Process B starts from the same checkpoint, emits a reconcile health snapshot before any new steady-state acquisition, and exits cleanly;
  • checkpoint generation and identity after Process B prove publication before checkpoint advance;
  • pressure/capacity fields appear in HEALTH_SNAPSHOT payloads;
  • no known_gap interval is marked complete unless a future A10 repair cutover evidence path proves it.

Process A - bounded acquire

Material output:

RC_A=0
user=mkt-transform
primary_group=marketdata
is_root=False
signal kind=HEALTH_SNAPSHOT ... checkpoint_identity=live-checkpoint-v1:sha256:dd0a2181d1c58c5c893ce072b01f841919f9bb6481ecb792db2394ce65dc0ce7,cycle=1,health=HEALTHY,phase=acquire,pressure_new_writes_allowed=True,pressure_state=NORMAL,session_state=CONTINUOUS
LIVE_INGEST_SERVER: STOPPED

Acceptance covered:

  • K02-compatible runtime identity (mkt-transform, non-root);
  • normal bounded acquire cycle reached HEALTHY;
  • checkpoint advanced only after the cycle's durable publication path;
  • K04/K05 capacity/pressure evidence appeared in the health signal.

Process B - restart/reconcile

Material output:

RC_B=0
user=mkt-transform
primary_group=marketdata
is_root=False
signal kind=HEALTH_SNAPSHOT ... checkpoint_identity=live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d,cycle=1,health=HEALTHY,phase=reconcile,pressure_new_writes_allowed=True,pressure_state=NORMAL,records=769,session_state=CONTINUOUS
LIVE_INGEST_SERVER: STOPPED

Acceptance covered:

  • restart used the existing checkpoint and entered phase=reconcile;
  • bounded reconciliation accepted 769 records and returned HEALTHY;
  • session stayed CONTINUOUS;
  • K04/K05 capacity/pressure evidence remained NORMAL;
  • stop was clean after the bounded proof cycle.

Final checkpoint

Material output:

{
  "checkpoint_identity": "live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d",
  "coverage_segment_id": "k10-restart-reconciliation-20260925-193111",
  "created_at": "2026-09-25T19:31:12.051818Z",
  "dataset_identity": {
    "dataset_kind": "trades",
    "instrument": "BTCUSDT",
    "layer": "canonical",
    "record_schema_id": "trade-v1",
    "venue": "bybit"
  },
  "generation": 2,
  "last_canonical_exchange_ts": "2026-09-25T19:31:08.815Z",
  "last_canonical_trade_id": "e7ea1e1d-bf73-581f-bf03-67adba52c9f0",
  "last_observed_sequence": "816124801187",
  "partition_key": "dt=2026-09-25",
  "revision": 2,
  "source_semantics_id": "bybit-public-trades-websocket-v1"
}

Acceptance covered:

  • checkpoint generation advanced from Process A to Process B (generation=2);
  • the final checkpoint binds the Bybit BTCUSDT canonical trade-v1 dataset;
  • the checkpoint binds the restart reconciliation segment and revision 2.

DataGateway readback

Command:

sudo -u mkt-transform env PGPASSFILE="$PGPASS" \
  CATALOG_DSN="host=127.0.0.1 port=5433 dbname=market_catalog user=market_catalog_writer" \
  CHECKPOINT="$CHECKPOINT" \
  /opt/market-platform/.venv/bin/python /tmp/lipr05-readback-check.py \
  | tee "$EVIDENCE_DIR/datagateway-readback-check.stdout"

Observed output:

{
  "catalog_partition_ids": [
    "189e296c-c71a-40dc-9b57-20f1a08303c2"
  ],
  "checkpoint_generation": 2,
  "checkpoint_identity": "live-checkpoint-v1:sha256:7cccafc4c27ce0c3bd1e532ae13acb210992fd1f99f8fcbfdde519df76ae175d",
  "checkpoint_last_trade": {
    "exchange_ts": "2026-09-25T19:31:08.815Z",
    "partition_key": "dt=2026-09-25",
    "sequence": "816124801187",
    "trade_id": "e7ea1e1d-bf73-581f-bf03-67adba52c9f0"
  },
  "checkpoint_revision": 2,
  "contains_checkpoint_anchor": true,
  "coverage_complete": false,
  "coverage_gaps": [
    {
      "end": "2026-09-25T19:31:08.816Z",
      "start": "2026-09-25T19:31:08.815001Z"
    }
  ],
  "duplicate_trade_key_count": 0,
  "rel_paths": [
    "dt=2026-09-25/part-193111.parquet"
  ],
  "request_end": "2026-09-25T19:31:08.816Z",
  "request_start": "2026-09-25T19:31:08.815Z",
  "rows_read": 4,
  "status": "PASS",
  "storage_root_ids": [
    "hot"
  ]
}

Acceptance covered:

  • DataGateway readback found the checkpoint anchor;
  • no duplicate canonical economic TradeKeyV1 was found in the readback slice;
  • metadata did not falsely claim complete coverage for an interval with an explicit uncovered tail (coverage_complete=false, coverage_gaps present);
  • readback stayed on the hot storage root and revision 2 restart partition.

Forced Long-Gap Evidence

Real forced long-gap execution was NOT EXECUTED on the target server. Issue #128 allows a forced long-gap scenario or an explicitly accepted equivalent simulation; the repository proof uses the equivalent hermetic simulation below so the target server is not intentionally degraded.

Equivalent hermetic coverage is currently:

  • tests.test_live_ingest_server_v1.test_gap_detected_on_startup_records_non_complete_gap_and_starts_new_segment;
  • tests.test_live_ingest_server_v1.test_open_gap_observability_survives_restart_after_checkpoint_advances;
  • tests.test_live_ingest_server_v1.test_sustained_gap_updates_one_open_gap_chain_instead_of_appending_events;
  • tests.test_live_gap_orchestration_v1.test_records_known_gap_without_partition_and_unproven_source_evaluation;
  • tests.test_live_gap_orchestration_v1.test_inconclusive_source_evaluation_is_retryable_and_not_unproven.

These tests prove the accepted equivalent simulation path: unresolved restart continuity records explicit non-complete known_gap coverage, preserves open gap observability across restart/checkpoint advance, supersedes repeated detections in one open chain, and keeps inconclusive source evaluation distinct from a completed negative result.

Acceptance State

Requirement State Evidence
Server runs under K02-compatible authority PASS Process A/B show user=mkt-transform, is_root=False
Publication-before-checkpoint during normal running PASS Process A health checkpoint plus final checkpoint generation
Publication-before-checkpoint during restart/reconcile PASS Process B phase=reconcile, final generation 2/revision 2
Bounded reconcile resumes when continuity is provable PASS Process B records=769, health=HEALTHY, session_state=CONTINUOUS
Long-gap remains explicit/non-complete PASS for equivalent simulation Hermetic tests listed above
No silent loss / no duplicate canonical economic trades PASS DataGateway readback contains_checkpoint_anchor=true, duplicate_trade_key_count=0
No false complete coverage PASS DataGateway readback coverage_complete=false with explicit coverage_gaps
Operator/runbook evidence covers session/publication/checkpoint/gap/repair/pressure/authority PASS Process logs, final checkpoint, readback JSON, hermetic gap tests
No secrets/bulk market data committed PASS This document records commands and bounded outputs only

Current Conclusion

Issue #128's production-readiness proof is satisfied for the bounded live-ingest server v1 path, assuming the documented hermetic long-gap tests are accepted as the issue's allowed equivalent simulation. The target server proved K02 runtime identity, bounded acquire, stop/restart, bounded reconcile, checkpoint generation advance, DataGateway readback of the checkpoint anchor, zero duplicate canonical economic trade keys in the readback slice, and no false complete coverage claim.