Live Ingest Server Deployment Runbook¶
Status: v1 deployment-local runbook for issue #135.
This runbook installs and proves the supervised Bybit BTCUSDT live-ingest
server operating path on homelab. It uses the already-owned
tools/live_ingest_server.py entrypoint and does not add ingestion,
publication, checkpoint, repair or trading semantics.
Boundaries¶
- Runs as
mkt-transform:marketdata. - Uses GitHub
originas the code authority. - Uses
/opt/market-platformas the stable deployed checkout. - Uses
/srv/marketdata/ storage root idhot. - Uses the existing PostgreSQL catalog connection through libpq environment and
PGPASSFILE; secrets stay deployment-local and out of Git. - Does not imply B06 live consumer cursors, K07 tier relocation, K09 retention and deletion authority, J08 live product mode, generic job runtime, strategy execution, order placement or multi-provider production.
Files¶
- Unit template:
infra/systemd/quant-platform-live-ingest.service - Non-secret env template:
infra/systemd/quant-platform-live-ingest.env.example - Target env file:
/etc/quant-platform/live-ingest-server.env - Target service file:
/etc/systemd/system/quant-platform-live-ingest.service - Target checkpoint:
/srv/marketdata/canonical/_k10_checkpoints/live-ingest-server-v1.json
Install Or Update¶
Run on homelab as the operator account:
cd /opt/market-platform
sudo -v
set -euo pipefail
# The code is promoted first, by the procedure of DECK_HANDOFF_RUNBOOK.md (Promotion): infra/promote-main.sh, clean tree,
# fast-forward only, on the branch main. This runbook never switches or merges branches: live-ingest runs the same commit as
# the API and the worker (ADR-0068 section 8), whatever that promotion moved the checkout to.
git status --short --branch
test -z "$(git status --porcelain)"
git remote get-url origin
test "$(git symbolic-ref --short -q HEAD)" = main
DEPLOYED_SHA="$(git rev-parse HEAD)"
echo "DEPLOYED_SHA=$DEPLOYED_SHA"
# /etc/quant-platform is shared with the API and worker units (DECK_HANDOFF_RUNBOOK.md), which run as users outside
# marketdata: root:root 0751 lets every unit's user reach its own files without listing the directory; the files
# keep their own owner and mode. Do not narrow it to root:marketdata 0750.
sudo install -d -m 0751 -o root -g root /etc/quant-platform
sudo install -m 0644 -o root -g root \
infra/systemd/quant-platform-live-ingest.service \
/etc/systemd/system/quant-platform-live-ingest.service
if [ ! -f /etc/quant-platform/live-ingest-server.env ]; then
sudo install -m 0640 -o root -g marketdata \
infra/systemd/quant-platform-live-ingest.env.example \
/etc/quant-platform/live-ingest-server.env
fi
sudo sed -i "s/^QP_LIVE_INGEST_CODE_REF=.*/QP_LIVE_INGEST_CODE_REF=${DEPLOYED_SHA}/" \
/etc/quant-platform/live-ingest-server.env
sudo -u mkt-transform test -r /home/mkt-transform/.pgpass
sudo -u mkt-transform test -w /srv/marketdata/canonical/_k10_checkpoints
sudo systemctl daemon-reload
sudo systemctl cat quant-platform-live-ingest.service
Before start, inspect /etc/quant-platform/live-ingest-server.env and correct
only deployment-local values. Do not add PGPASSWORD.
The unit runs under ProtectSystem=strict (the whole filesystem read-only
except ReadWritePaths=). If this deployment's QP_LIVE_INGEST_STORAGE_ROOT
is not /srv/marketdata, update ReadWritePaths= in the installed unit to
match before starting the service, or every checkpoint/publish write will
fail with EROFS/permission-denied.
Start, Stop, Restart And Status¶
sudo systemctl start quant-platform-live-ingest.service
sudo systemctl status --no-pager quant-platform-live-ingest.service
sudo journalctl -u quant-platform-live-ingest.service -n 80 --no-pager
sudo systemctl stop quant-platform-live-ingest.service
sudo systemctl status --no-pager quant-platform-live-ingest.service || true
sudo systemctl start quant-platform-live-ingest.service
sudo journalctl -u quant-platform-live-ingest.service -n 120 --no-pager
Expected material evidence:
- startup prints
=== effective runtime identity ===; - startup identity includes
user=mkt-transformandis_root=False; - logs include
signal kind=LIFECYCLE_TRANSITION; - logs include at least one
signal kind=HEALTH_SNAPSHOT; - after a restart, the first useful cycle reconciles from the durable checkpoint when one exists;
systemctl stopsends SIGTERM and the process exits throughLIVE_INGEST_SERVER: STOPPED.- no
EROFS/Permission deniedentries in the journal: this unit runs underProtectSystem=strict, so a storage-root or checkpoint-path mismatch withReadWritePaths=surfaces here, not as a Python traceback.
Readback Proof¶
After restart, read the checkpoint and run a narrow DataGateway check over the
last durable trade key. The check must prove the checkpoint anchor is present
and duplicate trade keys are absent. Record the JSON summary in
docs/integration/LIPR06A_SUPERVISED_LIVE_INGEST_DEPLOYMENT.md.
If readback reports partial coverage inside a half-open millisecond interval, that is acceptable only when the checkpoint anchor is present and duplicate keys are absent. Do not convert partial coverage into a false complete claim.
What The Server Publishes (ADR-0075 slices, DG-O O01)¶
Every publication is a slice: revision 1 of its own partition, never superseded. Under
canonical/trades/bybit/<INSTRUMENT>/trade-v1/:
dt=<UTC day of the window start>/seg=<HHMMSS.mmm of the window start>/part-001.parquet
/partition-manifest.json
/coverage-manifest.json
- The window is half-open
[start, end)in exchange time; the coverage assertion iscompleteover exactly that window. A window never crosses a UTC midnight. The trades of one millisecond are never split between two slices. - A
DataGatewayread of a live day returns every published trade (VALID_ONLY). A read that spans the pause between two cycles is refused understrictcoverage, because that pause is not covered (it ends with the continuous session,O02); underallow-partialit returns everything published. - The restart reconciliation publishes its records as a slice too, starting where the last published slice ended (continuity proven by the provider's recent-trades window). When that proven span reaches a UTC midnight before the first new trade, the span up to the midnight is a slice with no trade (a bridge), so the chain has no hole. A slice already published is never republished.
- "Already published" means valid: a slice that was sealed but never became valid (certification or eligibility failed) is not
readable and is not a replay. After the restart its trades are published again; if the window is the same, the failed attempt is
replaced by the next revision (
part-002.parquet, ...). A valid slice is never superseded. - A live trade before the end of what is already published is an ordering violation: the loop stops with the failure code
SLICE_ORDERING_VIOLATION, publishes nothing, and the next start reconciles from the checkpoint. - Partitions published before ADR-0075 (
dt=<day>, many superseded revisions) are left as they are; there is no repair.
Continuous session (ADR-0075 slices 9.2 and 9.3, O02 and O03): opt-in per unit¶
--mode continuous (environment variable QP_LIVE_INGEST_MODE) makes the server run ONE WebSocket session instead of the bounded
cycles: the connection is not closed between publications, a bounded buffer decouples the reader from the publisher, and slices
close after slice_seconds of wall clock or slice_max_trades trades. The default is cycles, the loop described above, and
nothing changes for a unit whose environment file does not set the mode (an empty or unset value is cycles, and an environment
file that predates these variables still starts the unit). Enabling it is an operator step: edit the environment file of the
instance and restart the unit; the repository enables nothing, and deployment of a changed server stays manual.
| Variable | Flag | Default | Meaning |
|---|---|---|---|
QP_LIVE_INGEST_MODE |
--mode |
cycles |
cycles or continuous |
QP_LIVE_INGEST_SLICE_SECONDS |
--slice-seconds |
300 | continuous only: wall-clock seconds after which an open slice closes |
QP_LIVE_INGEST_SLICE_MAX_TRADES |
--slice-max-trades |
50000 | continuous only: trades after which an open slice closes |
QP_LIVE_INGEST_BUFFER_MESSAGES |
--buffer-messages |
20000 | continuous only: bound of the reader's buffer; a full buffer ends the session |
The three bounds are valid only with continuous: with cycles the server refuses to start (exit status 2) rather than ignore
them, so leave them empty (an empty value is the default). A non-positive or non-integer value is refused before any connection.
--recent-limit also applies to the reconciliation after an interruption. The ping interval, pong timeout and reconnect backoff are
not configuration. --max-cycles, when used for a bounded proof, counts health snapshots: one per published slice in this mode.
Example, an instance switched to the continuous session with the defaults (/etc/quant-platform/live-ingest-server-ETHUSDT.env):
QP_LIVE_INGEST_MODE=continuous
QP_LIVE_INGEST_SLICE_SECONDS=
QP_LIVE_INGEST_SLICE_MAX_TRADES=
QP_LIVE_INGEST_BUFFER_MESSAGES=
What an operator sees:
- Startup prints
mode=continuousand the three bounds in force. - A disconnect, a missing pong, a full buffer or a trade below the open window ends the session; what was received is published,
the interruption is signalled (
HEALTH_SNAPSHOT,DEGRADED,phase=interruption, with the reason and the counters), the session reconnects with a bounded backoff, subscribes first and waits for the first trade of the new subscription. It then fetches the recent trades and reconciles from the last durable key; a subscription with no trade yet reconciles nothing and claims nothing. - Continuity proven (a trade of the subscription is in the recent-trades window): the reconciled trades are published as slices and
the chain stays exact. Anchor outside the window, or a window that does not meet the subscription: an explicit gap (ADR-0044),
recorded,
DEGRADED; a new window starts at the first trade after it and nothing is claimed across the gap. - Every published slice signals
phase=slicewithbuffer_high_waterandbuffer_capacity,slice_lag_ms,reconnectsandordering_violations. A UTC midnight with no trade is reported as an unclaimed span. A publication, catalog or checkpoint failure stops the loop (FAILURE), as in the cycle loop. - A
DataGatewayread understrictcoverage across an interruption's gap is refused; underallow-partialit returns everything published.
Rollback to the cycles: set QP_LIVE_INGEST_MODE=cycles (and the bounds empty), restart the unit. The checkpoint and the
published partitions are the same in both modes: the first cycle reconciles from the durable checkpoint.
One-hour real-provider proof (O03; the T02 proof for ETHUSDT)¶
Operator-executed; the repository starts nothing. It measures what ADR-0075 leaves as an assumption: the captured share against
the provider's archive, the ordering of the provider's exchange_ts, the slice sizes and the health signals. Use a disposable
catalog and storage root (or an ETHUSDT-only root the maintainer chooses): the proof never touches the BTCUSDT dataset or its
unit, and it enables nothing in production (the ADR-0074 section 6 capacity gate stands).
# 1. Run the continuous session for at least one hour, with the signals captured. Stop it cleanly with SIGTERM/SIGINT.
python tools/live_ingest_server.py --instrument ETHUSDT --mode continuous \
--storage-root <ROOT> --storage-root-id <ID> --checkpoint-path <ROOT>/canonical/_k10_checkpoints/live-ingest-server-v1-ETHUSDT.json \
--code-ref "$(git rev-parse HEAD)" 2>&1 | tee live-ethusdt-run.log
# 2. After the UTC day (or days) of the run has ended, fetch the archive of each day it touched (tools/fetch_bybit_trade_archive.py).
# 3. Produce the evidence (read-only; exit 0 = no stop condition, 1 = a stop condition, 2 = usage):
python tools/live_window_evidence.py --instrument ETHUSDT --storage-root <ROOT> \
--archive ETHUSDT<YYYY-MM-DD>.csv.gz [more days] \
--signals-log live-ethusdt-run.log --code-ref <SHA> --mode continuous > ethusdt-evidence.json
The report states, for the window compared: the archive trades in it, the trades a strict gateway read returns (chunks of at
most twelve slices, ending on slice boundaries), the captured ratio, the holes between exact chains with the archive trades each
contains and what declares it, the slices (count, rows and seconds: minimum, median, maximum) and the counters of the session.
Without --start/--end the window is the extent of the valid live slices; with them it is exactly the window asked for (a slice
that crosses a bound is refused, not clipped, and the archive of every day the whole window touches is required), so the archive
trades before the first slice and after the last count against the proof.
A missing trade in a hole is excused only by a span the run recorded that contains it: the known_gap assertion of an explicit
gap record under --storage-root (ADR-0044; its last durable key must lie in the segment before the hole) or an unclaimed_span
signal in the captured log. Each missing trade is classified against those spans, not against the hole around it: a gap that ends
before a later loss does not excuse that loss, and the end of a span is exclusive. The absence of a partition is not a declaration
of its cause.
Stop conditions (exit 1; do not read the proof as passed and report): an archive trade inside a claimed window that is not read
back (an undeclared loss); an archive trade in a hole outside every recorded gap span and signalled span (an undeclared loss); an archive
trade of the requested window before the first or after the last claimed window; a strict read of a claimed window refused; a trade
read twice or out of the canonical (exchange_ts, trade_id) order; a trade whose payload differs from the archive; valid slices that
overlap; a gap manifest that cannot be read. A shortfall against the archive inside a recorded span is reported, not hidden, and
is not a stop condition; an ordering violation is a measurement (it is an interruption by design), to be explained in the evidence.
Record the report, the log and the commit in the pull request of O03.
Rollback¶
This runbook does not automate rollback. If the service fails:
sudo systemctl stop quant-platform-live-ingest.service
sudo systemctl status --no-pager quant-platform-live-ingest.service || true
sudo journalctl -u quant-platform-live-ingest.service -n 200 --no-pager
Then inspect the repository and service environment manually. Do not run
git reset --hard, git clean, automated stash or pull-on-push deployment as
part of this runbook.
Additional Bybit Linear Instruments (ADR-0074, DG-N T02)¶
Status: artifacts and operator steps only. Nothing here is installed or enabled by the repository, and nothing here proves
two instruments running side by side or a real-provider run (parts 5 and 6 of T02).
ADR-0074 runs one live-ingest process per instrument.
BTCUSDT stays on quant-platform-live-ingest.service, with its own live-ingest-server.env and checkpoint, exactly as
above. The other instruments of the allow-list (ETHUSDT, SOLUSDT, XRPUSDT, DOGEUSDT) run as instances of one template
unit:
BTCUSDT (existing) |
any other instrument (template) | |
|---|---|---|
| Unit | quant-platform-live-ingest.service |
quant-platform-live-ingest@<INSTRUMENT>.service |
| Environment file | /etc/quant-platform/live-ingest-server.env |
/etc/quant-platform/live-ingest-server-<INSTRUMENT>.env |
| Checkpoint | QP_LIVE_INGEST_CHECKPOINT_PATH of the file |
${QP_LIVE_INGEST_CHECKPOINT_DIR}/live-ingest-server-v1-<INSTRUMENT>.json, built from the instance name |
--instrument |
omitted (defaults to BTCUSDT) |
--instrument %i, always on the command line |
What the template makes impossible, by construction:
- Forgetting the instrument. The instance name is the instrument; the command line is not read from the environment file.
- Two instances sharing a checkpoint. The checkpoint file name contains the instance name.
- Two writers on
BTCUSDT.ExecStartPrerefuses the instanceBTCUSDT; do not createlive-ingest-server-BTCUSDT.env. - An instrument outside the allow-list.
quant-platform-live-ingest@ADAUSDT.servicestarts and is refused by the server's own--instrumentoption (exit status 2). - A checkpoint of another instrument. If the file is there anyway, the loop ends with a
FAILUREsignal (CheckpointDomainMismatch) before it acquires or publishes anything.
Capacity gate (ADR-0074 section 6)¶
Enabling live ingest for more than BTCUSDT in production requires the Phase A measurements of #378: the projected time for
the hot storage root to reach the critical-pressure threshold of the live-ingest unit, at the measured rate of all enabled
instruments, must be at least 365 days. Until the measurement exists nothing beyond the bounded one-hour ETHUSDT proof
(part 6 of T02) is enabled. All instances share one storage root and one catalog; each one evaluates the pressure thresholds
of its own env file against the free space of the shared root, so the thresholds must be chosen for the sum of the enabled
instruments, not for one.
Add An Instance¶
Run on homelab as the operator account, after the code was promoted as above (DEPLOYED_SHA is the deployed commit):
cd /opt/market-platform
sudo -v
set -euo pipefail
INSTRUMENT=ETHUSDT # ETHUSDT | SOLUSDT | XRPUSDT | DOGEUSDT; never BTCUSDT
DEPLOYED_SHA="$(git rev-parse HEAD)"
test -z "$(git status --porcelain)"
sudo install -m 0644 -o root -g root \
infra/systemd/quant-platform-live-ingest@.service \
/etc/systemd/system/quant-platform-live-ingest@.service
if [ ! -f "/etc/quant-platform/live-ingest-server-${INSTRUMENT}.env" ]; then
sudo install -m 0640 -o root -g marketdata \
infra/systemd/quant-platform-live-ingest@.env.example \
"/etc/quant-platform/live-ingest-server-${INSTRUMENT}.env"
fi
sudo sed -i "s/^QP_LIVE_INGEST_CODE_REF=.*/QP_LIVE_INGEST_CODE_REF=${DEPLOYED_SHA}/" \
"/etc/quant-platform/live-ingest-server-${INSTRUMENT}.env"
sudo -u mkt-transform test -r /home/mkt-transform/.pgpass
sudo -u mkt-transform test -w /srv/marketdata/canonical/_k10_checkpoints
sudo systemctl daemon-reload
sudo systemctl cat "quant-platform-live-ingest@${INSTRUMENT}.service"
Inspect the instance's env file and correct only deployment-local values. Do not add PGPASSWORD. The hardening and
ReadWritePaths= notes above apply to the template unit as well.
Start, Stop And Status Of An Instance¶
sudo systemctl start "quant-platform-live-ingest@${INSTRUMENT}.service"
sudo systemctl status --no-pager "quant-platform-live-ingest@${INSTRUMENT}.service"
sudo journalctl -u "quant-platform-live-ingest@${INSTRUMENT}.service" -n 80 --no-pager
sudo systemctl stop "quant-platform-live-ingest@${INSTRUMENT}.service"
Expected evidence, in addition to the list above:
- the banner prints
instrument=<INSTRUMENT>; - every signal names the subject
bybit:<INSTRUMENT>:hot; - the checkpoint is
/srv/marketdata/canonical/_k10_checkpoints/live-ingest-server-v1-<INSTRUMENT>.json; - the instrument's data lands under
canonical/trades/bybit/<INSTRUMENT>/trade-v1/, and its explicit gap manifests in that directory only.
The instances are independent processes: stopping, restarting or failing one does not touch another. A history gap is handled
as ADR-0044 decides (explicit, no repair source); the period before an instance's first start is not acquired and is never
reported complete. Historical days come from the archive import (BYBIT_ARCHIVE_IMPORT_RUNBOOK.md), which refuses a day that
already has a partition.
Remove An Instance¶
sudo systemctl stop "quant-platform-live-ingest@${INSTRUMENT}.service"
sudo systemctl disable "quant-platform-live-ingest@${INSTRUMENT}.service"
Data, checkpoint and env file are left in place; deleting them is the operator's decision, not this runbook's.