unrip/archive/implementation/20260612T150844Z-quote-lifecycle-retention-and-analytics-rollups-proof.md

Implementation Proof: Quote lifecycle retention and analytics rollups

Status: open Opened: 2026-06-06

Target outcome

The system must keep durable successful trade evidence and useful competitiveness analytics without allowing quote firehose or non-success lifecycle detail to fill the Postgres PVC again.

This turn succeeds when raw quotes, normalized quote demand, rejected decisions, failed executor results, and not-filled outcomes have explicit DB-backed retention and compact rollups, while completed trade evidence remains durable and operator-visible.

Why this is the next architecture test

The maker timing turn made response behavior explainable, but live operation exposed a storage false path:

• raw quote retention alone was not enough
• trade_decisions and swap_demand_events grew into the largest tables
• emergency cleanup had to prune almost all non-success quote lifecycle rows
• the operator still needs competitiveness truth without storing every quote forever

The next architecture test is whether unrip can keep the right evidence permanently and summarize the rest before deleting it.

Scope

• Define quote-history data classes:
  • successful settled trade evidence
  • in-flight lifecycle detail
  • non-success detail
  • raw debug firehose
  • aggregate analytics
• Keep successful trade evidence permanently:
  • completed quote outcomes
  • linked normalized demand, decision, command, executor result, and attribution rows
  • inventory/settlement evidence needed to explain the trade
  • config identifiers and payload fields needed to reproduce the decision
• Add compact rollups before pruning detail:
  • pair
  • direction
  • request kind
  • edge bps
  • notional bucket
  • quote-age bucket
  • result code
  • failure category
  • outcome status
  • counts and timing bucket summaries
• Add DB-backed retention policy for quote lifecycle data, not env vars.
• Enforce bounded retention in history-writer or a repo-owned maintenance path.
• Add storage guardrails that detect table/PVC pressure and tighten non-success detail retention without deleting successful trade evidence.
• Surface retention mode, table sizes, retained successful rows, and rollup freshness in the operator dashboard.
• Keep old rows readable and dashboards tolerant of missing rollups.
• Deploy only through repo workflow.

Assumptions

• Completed quote outcome attribution is the current durable proof of successful trade settlement.
• Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
• Non-success quote detail is useful for short live debugging, but long-term analysis can use rollups plus sampled/recent detail.
• Postgres remains the canonical event store for the current system.
• The current single-node cluster has enough disk for compact rollups and successful evidence, but not unbounded quote detail.

Turn-shaping rules

• Do not change strategy selection or maker response behavior.
• Do not skip quotes because of storage or relay-error concerns.
• Do not loosen edge, notional, inventory, arming, signer, or pair enablement.
• No active pair, edge, notional, retention, or latency env vars.
• Do not delete completed trade evidence.
• Do not label rollup counts as realized PnL or settled trades.
• Manual emergency pruning may be documented as evidence, but the lasting fix must be repo-owned and deployed from push.
• Destructive cleanup paths must have dry-run/test coverage and must fail closed on missing preserve criteria.

Non-goals

• No new trading strategy.
• No response-age or competitiveness-based quote skipping.
• No new venue.
• No broad data warehouse outside Postgres.
• No fee-complete realized PnL.
• No permanent infrastructure spend unless explicitly approved.
• No retroactive recovery of quote detail already pruned during emergency cleanup.

Required operator behavior

Storage Truth

The operator must be able to see:

• current size of quote lifecycle tables
• estimated row counts by table family
• retention policy in effect
• last prune time and prune result
• rollup freshness
• count of permanent successful trade rows
• whether the system is in normal or pressure retention mode

Analytics Truth

The operator must still be able to answer after detail pruning:

• how many quotes arrived by pair/direction/request kind
• how many were actionable, rejected, submitted, failed, not filled, or completed
• whether failures cluster by quote-age bucket or notional bucket
• whether quote_not_found_or_finished dominates a pair/direction
• how many accepted responses became completed settlement evidence

Evidence Truth

For completed trades, the operator must retain per-quote lifecycle detail, not only rollups.

For non-success cases, old detail may disappear, but the rollup must preserve aggregate counts and bucketed timing truth.

Semantic invariants

• Completed settlement evidence is permanent unless the operator explicitly approves an archive/export workflow in a future turn.
• In-flight rows inside the retention window are not pruned before attribution can observe settlement/no-fill.
• Rollups are written before detail deletion.
• Retention policy is DB-owned and visible.
• Missing or invalid retention policy does not delete data.
• Pressure mode may shorten non-success detail retention, but never deletes successful evidence.
• Pair-native labels remain canonical.
• Old rows without new retention metadata remain readable.

Definition of done

• Quote lifecycle retention policy exists in DB-owned config or an equivalent repo-owned persisted mechanism.
• Rollup tables or records exist for maker quote lifecycle analytics.
• History-writer or a repo-owned maintenance service writes rollups before pruning detail.
• Non-success detail retention prevents raw_near_intents_quotes, swap_demand_events, trade_decisions, execute_trade_commands, trade_execution_results, and quote_outcome_attributions from unbounded growth.
• Successful completed quote lifecycle rows are preserved across retention.
• Storage guardrails surface pressure and tighten only non-success detail.
• Dashboard exposes storage/retention/rollup status.
• Tests cover rollup generation, retention preserve rules, pressure mode, old-row compatibility, and dashboard visibility.
• Full npm test passes.
• Dashboard bundle build passes.
• The result is deployed through repo workflow.
• Live post-deploy evidence shows table sizes stay bounded while quote ingest and responding continue.

Validation evidence required

• Unit tests for identifying successful quote ids to preserve.
• Unit tests for rollup grouping by pair, direction, request kind, result code, failure category, quote-age bucket, notional bucket, and outcome status.
• Tests proving rollups are written before detail pruning.
• Tests proving completed successful lifecycle rows survive pruning.
• Tests proving non-success detail older than the configured window is pruned.
• Tests proving missing/invalid retention policy fails closed.
• Tests proving pressure mode cannot delete completed evidence.
• Dashboard/static tests for storage and retention surfaces.
• Static deployment tests proving no retention/pair/strategy env vars are added.
• Full npm test.
• Dashboard bundle build.
• Live evidence after deployment:
  • table sizes
  • latest prune events
  • latest rollup timestamps
  • successful trade row count
  • recent quote flow still writing rows

Failure conditions

• The DB can still fill from non-success quote lifecycle rows under normal live quote volume.
• Rollups are missing or stale while detail is pruned.
• Completed settlement evidence is deleted.
• Retention behavior depends on env vars or untracked manual SQL.
• The implementation changes maker strategy behavior or skips quotes.
• The dashboard hides storage pressure or makes retained rollups look like per-quote evidence.
• Old rows become unreadable.

Current real before this turn

• Maker timing, relay failure classification, competitiveness summaries, and lifecycle dashboards are deployed.
• Emergency cleanup on 2026-06-05 proved the largest growth was normalized quote/decision history, not raw quotes alone.
• A short-term deployed prune now deletes old non-success lifecycle rows and preserves completed quote outcomes.
• Live evidence after cleanup showed quote tables reduced to MB scale while quote flow resumed.
• The current fix is functional but not yet a full retention product: rollups, storage dashboard truth, policy introspection, and pressure guardrails still need to be made first-class.

Deliberately not built by this proof

• Recovery of already-pruned historical non-success quote detail.
• Fee-complete realized PnL.
• Venue-native terminal fill ids.
• External archival warehouse.
• Automatic strategy changes based on rollups.