# Implementation Proof: Quote lifecycle retention and analytics rollups

Status: open
Opened: 2026-06-06

## Target outcome
The system must keep durable successful trade evidence and useful competitiveness analytics without allowing quote firehose or non-success lifecycle detail to fill the Postgres PVC again.

This turn succeeds when raw quotes, normalized quote demand, rejected decisions, failed executor results, and not-filled outcomes have explicit DB-backed retention and compact rollups, while completed trade evidence remains durable and operator-visible.

## Why this is the next architecture test
The maker timing turn made response behavior explainable, but live operation exposed a storage false path:

- raw quote retention alone was not enough
- `trade_decisions` and `swap_demand_events` grew into the largest tables
- emergency cleanup had to prune almost all non-success quote lifecycle rows
- the operator still needs competitiveness truth without storing every quote forever

The next architecture test is whether `unrip` can keep the right evidence permanently and summarize the rest before deleting it.

## Scope
- Define quote-history data classes:
  - successful settled trade evidence
  - in-flight lifecycle detail
  - non-success detail
  - raw debug firehose
  - aggregate analytics
- Keep successful trade evidence permanently:
  - completed quote outcomes
  - linked normalized demand, decision, command, executor result, and attribution rows
  - inventory/settlement evidence needed to explain the trade
  - config identifiers and payload fields needed to reproduce the decision
- Add compact rollups before pruning detail:
  - pair
  - direction
  - request kind
  - edge bps
  - notional bucket
  - quote-age bucket
  - result code
  - failure category
  - outcome status
  - counts and timing bucket summaries
- Add DB-backed retention policy for quote lifecycle data, not env vars.
- Enforce bounded retention in history-writer or a repo-owned maintenance path.
- Add storage guardrails that detect table/PVC pressure and tighten non-success detail retention without deleting successful trade evidence.
- Surface retention mode, table sizes, retained successful rows, and rollup freshness in the operator dashboard.
- Keep old rows readable and dashboards tolerant of missing rollups.
- Deploy only through repo workflow.

## Assumptions
- Completed quote outcome attribution is the current durable proof of successful trade settlement.
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
- Non-success quote detail is useful for short live debugging, but long-term analysis can use rollups plus sampled/recent detail.
- Postgres remains the canonical event store for the current system.
- The current single-node cluster has enough disk for compact rollups and successful evidence, but not unbounded quote detail.

## Turn-shaping rules
- Do not change strategy selection or maker response behavior.
- Do not skip quotes because of storage or relay-error concerns.
- Do not loosen edge, notional, inventory, arming, signer, or pair enablement.
- No active pair, edge, notional, retention, or latency env vars.
- Do not delete completed trade evidence.
- Do not label rollup counts as realized PnL or settled trades.
- Manual emergency pruning may be documented as evidence, but the lasting fix must be repo-owned and deployed from push.
- Destructive cleanup paths must have dry-run/test coverage and must fail closed on missing preserve criteria.

## Non-goals
- No new trading strategy.
- No response-age or competitiveness-based quote skipping.
- No new venue.
- No broad data warehouse outside Postgres.
- No fee-complete realized PnL.
- No permanent infrastructure spend unless explicitly approved.
- No retroactive recovery of quote detail already pruned during emergency cleanup.

## Required operator behavior

### Storage Truth
The operator must be able to see:

- current size of quote lifecycle tables
- estimated row counts by table family
- retention policy in effect
- last prune time and prune result
- rollup freshness
- count of permanent successful trade rows
- whether the system is in normal or pressure retention mode

### Analytics Truth
The operator must still be able to answer after detail pruning:

- how many quotes arrived by pair/direction/request kind
- how many were actionable, rejected, submitted, failed, not filled, or completed
- whether failures cluster by quote-age bucket or notional bucket
- whether `quote_not_found_or_finished` dominates a pair/direction
- how many accepted responses became completed settlement evidence

### Evidence Truth
For completed trades, the operator must retain per-quote lifecycle detail, not only rollups.

For non-success cases, old detail may disappear, but the rollup must preserve aggregate counts and bucketed timing truth.

## Semantic invariants
- Completed settlement evidence is permanent unless the operator explicitly approves an archive/export workflow in a future turn.
- In-flight rows inside the retention window are not pruned before attribution can observe settlement/no-fill.
- Rollups are written before detail deletion.
- Retention policy is DB-owned and visible.
- Missing or invalid retention policy does not delete data.
- Pressure mode may shorten non-success detail retention, but never deletes successful evidence.
- Pair-native labels remain canonical.
- Old rows without new retention metadata remain readable.

## Definition of done
- Quote lifecycle retention policy exists in DB-owned config or an equivalent repo-owned persisted mechanism.
- Rollup tables or records exist for maker quote lifecycle analytics.
- History-writer or a repo-owned maintenance service writes rollups before pruning detail.
- Non-success detail retention prevents `raw_near_intents_quotes`, `swap_demand_events`, `trade_decisions`, `execute_trade_commands`, `trade_execution_results`, and `quote_outcome_attributions` from unbounded growth.
- Successful completed quote lifecycle rows are preserved across retention.
- Storage guardrails surface pressure and tighten only non-success detail.
- Dashboard exposes storage/retention/rollup status.
- Tests cover rollup generation, retention preserve rules, pressure mode, old-row compatibility, and dashboard visibility.
- Full `npm test` passes.
- Dashboard bundle build passes.
- The result is deployed through repo workflow.
- Live post-deploy evidence shows table sizes stay bounded while quote ingest and responding continue.

## Validation evidence required
- Unit tests for identifying successful quote ids to preserve.
- Unit tests for rollup grouping by pair, direction, request kind, result code, failure category, quote-age bucket, notional bucket, and outcome status.
- Tests proving rollups are written before detail pruning.
- Tests proving completed successful lifecycle rows survive pruning.
- Tests proving non-success detail older than the configured window is pruned.
- Tests proving missing/invalid retention policy fails closed.
- Tests proving pressure mode cannot delete completed evidence.
- Dashboard/static tests for storage and retention surfaces.
- Static deployment tests proving no retention/pair/strategy env vars are added.
- Full `npm test`.
- Dashboard bundle build.
- Live evidence after deployment:
  - table sizes
  - latest prune events
  - latest rollup timestamps
  - successful trade row count
  - recent quote flow still writing rows

## Failure conditions
- The DB can still fill from non-success quote lifecycle rows under normal live quote volume.
- Rollups are missing or stale while detail is pruned.
- Completed settlement evidence is deleted.
- Retention behavior depends on env vars or untracked manual SQL.
- The implementation changes maker strategy behavior or skips quotes.
- The dashboard hides storage pressure or makes retained rollups look like per-quote evidence.
- Old rows become unreadable.

## Current real before this turn
- Maker timing, relay failure classification, competitiveness summaries, and lifecycle dashboards are deployed.
- Emergency cleanup on 2026-06-05 proved the largest growth was normalized quote/decision history, not raw quotes alone.
- A short-term deployed prune now deletes old non-success lifecycle rows and preserves completed quote outcomes.
- Live evidence after cleanup showed quote tables reduced to MB scale while quote flow resumed.
- The current fix is functional but not yet a full retention product: rollups, storage dashboard truth, policy introspection, and pressure guardrails still need to be made first-class.

## Deliberately not built by this proof
- Recovery of already-pruned historical non-success quote detail.
- Fee-complete realized PnL.
- Venue-native terminal fill ids.
- External archival warehouse.
- Automatic strategy changes based on rollups.