8d2ca61fa1
Proof: Archived the quote lifecycle retention turn and opened detailed PROOF.md/IMPLEMENTATION.md for removing verifier salt refresh from quote-response execution hot path. Assumptions: The current retention turn should be paused so the live salt-induced executor queue latency defect can take priority; no backlog item was selected because the turn was opened directly from live operator evidence. Still fake: Planning only; salt cache refactor, executor rejection behavior, health/dashboard surfaces, deployment, and live latency evidence are not implemented yet.
168 lines
8.7 KiB
Markdown
168 lines
8.7 KiB
Markdown
# Implementation Proof: Quote lifecycle retention and analytics rollups
|
|
|
|
Status: open
|
|
Opened: 2026-06-06
|
|
|
|
## Target outcome
|
|
The system must keep durable successful trade evidence and useful competitiveness analytics without allowing quote firehose or non-success lifecycle detail to fill the Postgres PVC again.
|
|
|
|
This turn succeeds when raw quotes, normalized quote demand, rejected decisions, failed executor results, and not-filled outcomes have explicit DB-backed retention and compact rollups, while completed trade evidence remains durable and operator-visible.
|
|
|
|
## Why this is the next architecture test
|
|
The maker timing turn made response behavior explainable, but live operation exposed a storage false path:
|
|
|
|
- raw quote retention alone was not enough
|
|
- `trade_decisions` and `swap_demand_events` grew into the largest tables
|
|
- emergency cleanup had to prune almost all non-success quote lifecycle rows
|
|
- the operator still needs competitiveness truth without storing every quote forever
|
|
|
|
The next architecture test is whether `unrip` can keep the right evidence permanently and summarize the rest before deleting it.
|
|
|
|
## Scope
|
|
- Define quote-history data classes:
|
|
- successful settled trade evidence
|
|
- in-flight lifecycle detail
|
|
- non-success detail
|
|
- raw debug firehose
|
|
- aggregate analytics
|
|
- Keep successful trade evidence permanently:
|
|
- completed quote outcomes
|
|
- linked normalized demand, decision, command, executor result, and attribution rows
|
|
- inventory/settlement evidence needed to explain the trade
|
|
- config identifiers and payload fields needed to reproduce the decision
|
|
- Add compact rollups before pruning detail:
|
|
- pair
|
|
- direction
|
|
- request kind
|
|
- edge bps
|
|
- notional bucket
|
|
- quote-age bucket
|
|
- result code
|
|
- failure category
|
|
- outcome status
|
|
- counts and timing bucket summaries
|
|
- Add DB-backed retention policy for quote lifecycle data, not env vars.
|
|
- Enforce bounded retention in history-writer or a repo-owned maintenance path.
|
|
- Add storage guardrails that detect table/PVC pressure and tighten non-success detail retention without deleting successful trade evidence.
|
|
- Surface retention mode, table sizes, retained successful rows, and rollup freshness in the operator dashboard.
|
|
- Keep old rows readable and dashboards tolerant of missing rollups.
|
|
- Deploy only through repo workflow.
|
|
|
|
## Assumptions
|
|
- Completed quote outcome attribution is the current durable proof of successful trade settlement.
|
|
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
|
|
- Non-success quote detail is useful for short live debugging, but long-term analysis can use rollups plus sampled/recent detail.
|
|
- Postgres remains the canonical event store for the current system.
|
|
- The current single-node cluster has enough disk for compact rollups and successful evidence, but not unbounded quote detail.
|
|
|
|
## Turn-shaping rules
|
|
- Do not change strategy selection or maker response behavior.
|
|
- Do not skip quotes because of storage or relay-error concerns.
|
|
- Do not loosen edge, notional, inventory, arming, signer, or pair enablement.
|
|
- No active pair, edge, notional, retention, or latency env vars.
|
|
- Do not delete completed trade evidence.
|
|
- Do not label rollup counts as realized PnL or settled trades.
|
|
- Manual emergency pruning may be documented as evidence, but the lasting fix must be repo-owned and deployed from push.
|
|
- Destructive cleanup paths must have dry-run/test coverage and must fail closed on missing preserve criteria.
|
|
|
|
## Non-goals
|
|
- No new trading strategy.
|
|
- No response-age or competitiveness-based quote skipping.
|
|
- No new venue.
|
|
- No broad data warehouse outside Postgres.
|
|
- No fee-complete realized PnL.
|
|
- No permanent infrastructure spend unless explicitly approved.
|
|
- No retroactive recovery of quote detail already pruned during emergency cleanup.
|
|
|
|
## Required operator behavior
|
|
|
|
### Storage Truth
|
|
The operator must be able to see:
|
|
|
|
- current size of quote lifecycle tables
|
|
- estimated row counts by table family
|
|
- retention policy in effect
|
|
- last prune time and prune result
|
|
- rollup freshness
|
|
- count of permanent successful trade rows
|
|
- whether the system is in normal or pressure retention mode
|
|
|
|
### Analytics Truth
|
|
The operator must still be able to answer after detail pruning:
|
|
|
|
- how many quotes arrived by pair/direction/request kind
|
|
- how many were actionable, rejected, submitted, failed, not filled, or completed
|
|
- whether failures cluster by quote-age bucket or notional bucket
|
|
- whether `quote_not_found_or_finished` dominates a pair/direction
|
|
- how many accepted responses became completed settlement evidence
|
|
|
|
### Evidence Truth
|
|
For completed trades, the operator must retain per-quote lifecycle detail, not only rollups.
|
|
|
|
For non-success cases, old detail may disappear, but the rollup must preserve aggregate counts and bucketed timing truth.
|
|
|
|
## Semantic invariants
|
|
- Completed settlement evidence is permanent unless the operator explicitly approves an archive/export workflow in a future turn.
|
|
- In-flight rows inside the retention window are not pruned before attribution can observe settlement/no-fill.
|
|
- Rollups are written before detail deletion.
|
|
- Retention policy is DB-owned and visible.
|
|
- Missing or invalid retention policy does not delete data.
|
|
- Pressure mode may shorten non-success detail retention, but never deletes successful evidence.
|
|
- Pair-native labels remain canonical.
|
|
- Old rows without new retention metadata remain readable.
|
|
|
|
## Definition of done
|
|
- Quote lifecycle retention policy exists in DB-owned config or an equivalent repo-owned persisted mechanism.
|
|
- Rollup tables or records exist for maker quote lifecycle analytics.
|
|
- History-writer or a repo-owned maintenance service writes rollups before pruning detail.
|
|
- Non-success detail retention prevents `raw_near_intents_quotes`, `swap_demand_events`, `trade_decisions`, `execute_trade_commands`, `trade_execution_results`, and `quote_outcome_attributions` from unbounded growth.
|
|
- Successful completed quote lifecycle rows are preserved across retention.
|
|
- Storage guardrails surface pressure and tighten only non-success detail.
|
|
- Dashboard exposes storage/retention/rollup status.
|
|
- Tests cover rollup generation, retention preserve rules, pressure mode, old-row compatibility, and dashboard visibility.
|
|
- Full `npm test` passes.
|
|
- Dashboard bundle build passes.
|
|
- The result is deployed through repo workflow.
|
|
- Live post-deploy evidence shows table sizes stay bounded while quote ingest and responding continue.
|
|
|
|
## Validation evidence required
|
|
- Unit tests for identifying successful quote ids to preserve.
|
|
- Unit tests for rollup grouping by pair, direction, request kind, result code, failure category, quote-age bucket, notional bucket, and outcome status.
|
|
- Tests proving rollups are written before detail pruning.
|
|
- Tests proving completed successful lifecycle rows survive pruning.
|
|
- Tests proving non-success detail older than the configured window is pruned.
|
|
- Tests proving missing/invalid retention policy fails closed.
|
|
- Tests proving pressure mode cannot delete completed evidence.
|
|
- Dashboard/static tests for storage and retention surfaces.
|
|
- Static deployment tests proving no retention/pair/strategy env vars are added.
|
|
- Full `npm test`.
|
|
- Dashboard bundle build.
|
|
- Live evidence after deployment:
|
|
- table sizes
|
|
- latest prune events
|
|
- latest rollup timestamps
|
|
- successful trade row count
|
|
- recent quote flow still writing rows
|
|
|
|
## Failure conditions
|
|
- The DB can still fill from non-success quote lifecycle rows under normal live quote volume.
|
|
- Rollups are missing or stale while detail is pruned.
|
|
- Completed settlement evidence is deleted.
|
|
- Retention behavior depends on env vars or untracked manual SQL.
|
|
- The implementation changes maker strategy behavior or skips quotes.
|
|
- The dashboard hides storage pressure or makes retained rollups look like per-quote evidence.
|
|
- Old rows become unreadable.
|
|
|
|
## Current real before this turn
|
|
- Maker timing, relay failure classification, competitiveness summaries, and lifecycle dashboards are deployed.
|
|
- Emergency cleanup on 2026-06-05 proved the largest growth was normalized quote/decision history, not raw quotes alone.
|
|
- A short-term deployed prune now deletes old non-success lifecycle rows and preserves completed quote outcomes.
|
|
- Live evidence after cleanup showed quote tables reduced to MB scale while quote flow resumed.
|
|
- The current fix is functional but not yet a full retention product: rollups, storage dashboard truth, policy introspection, and pressure guardrails still need to be made first-class.
|
|
|
|
## Deliberately not built by this proof
|
|
- Recovery of already-pruned historical non-success quote detail.
|
|
- Fee-complete realized PnL.
|
|
- Venue-native terminal fill ids.
|
|
- External archival warehouse.
|
|
- Automatic strategy changes based on rollups.
|