unrip/archive/implementation/20260612T150844Z-quote-lifecycle-retention-and-analytics-rollups-implementation.md

# Implementation Turn: Quote lifecycle retention and analytics rollups

Status: open
Opened: 2026-06-06

## Goal
Keep durable successful trade evidence while preventing quote firehose and non-success lifecycle rows from filling Postgres by adding explicit retention policy, compact rollups, table-size guardrails, and operator-visible storage truth.

## Selected backlog items
- none selected; this turn was opened directly from the operator request on 2026-06-06.

## Design rules
- This turn is storage and analytics only.
- Do not change strategy actionability, quote response policy, edge thresholds, inventory checks, pair enablement, arming, signer checks, or relay submission behavior.
- Do not skip quotes to avoid relay errors.
- DB config and persisted runtime state are canonical; do not add active retention env vars.
- Completed successful trade evidence must survive every cleanup path.
- Non-success quote detail may be short-lived if aggregate rollups are persisted first.
- Old rows must remain readable.
- Deploy only through repo workflow.

## Problem statement
The previous raw quote retention fix did not prevent Postgres growth because normalized lifecycle tables became the dominant storage users:

- `trade_decisions`
- `swap_demand_events`
- `execute_trade_commands`
- `trade_execution_results`
- `quote_outcome_attributions`

Emergency cleanup proved the system can keep completed quote outcomes and linked rows while pruning non-success detail, but that cleanup was operational, not a complete product path. The repository now needs a durable retention and rollup layer so live quote analytics remain useful without unbounded row accumulation.

## Backend changes

### 1. Inspect current storage and retention paths
- Inspect:
  - `ensureHistorySchema`
  - quote lifecycle table creation and indexes
  - history-writer batch ingestion
  - current raw quote prune
  - current non-success lifecycle prune
  - dashboard bootstrap queries
  - competitiveness aggregation helpers
  - runtime health and alert summaries
  - deploy/static config tests
- Identify all tables that grow with quote firehose or lifecycle activity.
- Record current row sizes and indexes in implementation notes or test fixtures where useful.

### 2. Define retention data classes
- Add a shared retention classifier for quote-linked rows:
  - `successful_trade_evidence`
  - `in_flight_detail`
  - `non_success_detail`
  - `raw_debug_firehose`
  - `aggregate_rollup`
- Preserve successful evidence when:
  - `quote_outcome_attributions.outcome_status = 'completed'`, or
  - attribution status indicates successful linkage such as `heuristic_match`, `exact_match`, or a locally established equivalent.
- Treat submitted/not_filled/failed/rejected rows as non-success detail once outside the retention window.
- Keep classification pair-native and quote-id based.
- Add tests for classification using current and legacy payload shapes.

### 3. DB-backed retention policy
- Add DB-owned retention policy storage, either:
  - a narrow `retention_policies` table, or
  - a versioned config row if an established local config pattern fits better.
- Candidate fields:
  - `policy_key`
  - `enabled`
  - `normal_detail_retention_ms`
  - `pressure_detail_retention_ms`
  - `raw_retention_ms`
  - `rollup_granularity_ms`
  - `rollup_retention_ms`
  - `preserve_successful_evidence`
  - `max_delete_rows_per_pass`
  - `pressure_table_bytes`
  - `pressure_database_bytes`
  - `updated_at`
  - `config_version`
- Seed conservative defaults in repo-owned DB seed/migration code.
- Missing or invalid policy must fail closed by skipping prune, not by deleting data.
- No env var fallback.

### 4. Rollup model
- Add compact quote lifecycle rollups before pruning detail.
- Store rollup rows keyed by:
  - rollup window start/end
  - pair id
  - direction
  - request kind
  - edge bps bucket or exact configured edge
  - notional asset
  - notional bucket
  - quote-age bucket
  - result code
  - failure category
  - outcome status
- Store compact measures:
  - quote count
  - decision count
  - command count
  - executor result count
  - relay accepted count
  - relay failed count
  - not_filled count
  - completed count
  - p50/p90-compatible timing bucket counts, or fixed timing histogram buckets
  - min/max observed timestamps
  - source row high-water mark or computed-at watermark
- Prefer bucket histograms over storing raw samples indefinitely.
- Add indexes for dashboard rollup queries.

### 5. Rollup writer
- Extend history-writer or add a narrow repo-owned maintenance module used by history-writer.
- On a schedule:
  - find recent unrolled quote lifecycle rows
  - aggregate them into rollup windows
  - upsert rollup rows idempotently
  - record rollup watermarks
- Rollup must run before detail prune.
- If rollup fails, detail prune for the affected window must not run.
- Add tests for idempotent upsert and no double-counting.

### 6. Detail retention prune
- Replace ad hoc non-success lifecycle prune with policy-driven retention.
- Prune these quote-linked detail tables only after successful rollup:
  - `raw_near_intents_quotes`
  - `swap_demand_events`
  - `trade_decisions`
  - `execute_trade_commands`
  - `trade_execution_results`
  - `quote_outcome_attributions`
- Preserve:
  - all successful quote ids and linked lifecycle rows
  - rows within the active retention window
  - rows whose outcome attribution is still in flight and younger than the configured outcome grace window
- Delete in bounded batches and expose per-table counts.
- Prefer table/index bloat control using repo-owned `VACUUM`/`REINDEX` guidance or tested maintenance where appropriate.
- Do not use untracked manual SQL as the normal path.

### 7. Pressure guardrails
- Add DB/table size inspection helpers using Postgres catalog views.
- Add a retention mode decision:
  - `normal`
  - `pressure`
  - `blocked_invalid_policy`
  - `blocked_rollup_stale`
- In pressure mode:
  - shorten non-success detail retention to the DB policy's pressure value
  - keep successful evidence
  - keep rollups
  - log and expose the pressure reason
- Do not disarm, pause, or alter strategy behavior from storage pressure in this turn unless an existing safety invariant already does so.
- Add tests proving pressure mode does not delete successful evidence.

### 8. Operator dashboard
- Add a storage/retention section to the operator dashboard.
- Show:
  - table sizes and approximate row counts
  - retention mode
  - configured retention windows
  - last rollup time
  - last prune time
  - rows pruned by table
  - successful quote evidence count
  - rollup row count and freshness
- Update competitiveness views to use rollups for older windows and recent detail for live drilldown.
- Mark old pruned detail plainly as unavailable rather than empty.
- Keep pair-native labels.
- Add dashboard tests.

### 9. Runtime health and alerts
- Add runtime health fields for:
  - retention mode
  - storage pressure
  - rollup freshness
  - prune failures
  - table size thresholds
- Add warning alerts for:
  - rollup stale beyond threshold
  - retention blocked by invalid policy
  - quote lifecycle tables above pressure threshold
- Do not make storage pressure look like a trading result or settled PnL issue.
- Add tests for alert payloads and pair-independent storage warnings.

### 10. Repo-owned emergency command
- Add or extend an ops script for dry-run and execute retention maintenance.
- Required behavior:
  - dry-run default
  - show preserve count for successful quote ids
  - show delete candidates by table
  - refuse to run if preserve query fails
  - refuse to run if rollups are stale unless explicitly marked as emergency and documented
- This script is for controlled recovery, not normal operation.

## Data and persistence
- Use additive schema changes where practical.
- Keep successful lifecycle rows in existing tables so old dashboard drilldowns continue to work.
- Keep rollups compact enough to retain for substantially longer than detail.
- Keep raw JSON payloads only in short-lived detail or successful evidence rows.
- Store original relay failure category/text in rollup dimensions where already normalized.
- Maintain old-row compatibility for missing timing/retention fields.

## Edge cases
- Missing quote id: do not classify as successful evidence; retain only inside short recent window unless rollup logic can safely count it.
- Missing outcome attribution table or failed preserve query: skip prune.
- Rollup watermark missing: skip prune for affected detail window.
- Clock skew or missing timestamps: bucket as `unknown`, do not invent age.
- Duplicate event ids: rollup idempotency must not double count.
- Completed quote has no raw row because raw was already pruned: preserve available linked lifecycle rows.
- Existing emergency-pruned history cannot be recreated.
- Old rows with legacy BTC/EURe fields remain readable and do not break rollups.

## Concrete implementation order

### Phase 1. Storage audit and contracts
- Inspect schema, indexes, history-writer retention, dashboard loaders, and runtime-health paths.
- Add retention classifier helpers and tests.
- Add storage metrics helper tests using SQL/static fixtures.

### Phase 2. Retention policy
- Add DB-backed retention policy schema/seed.
- Add policy loader and validation.
- Add fail-closed tests for missing/invalid policy.

### Phase 3. Rollup storage and writer
- Add rollup table/schema.
- Add rollup aggregation helper and SQL upsert path.
- Add idempotency and grouping tests for nBTC/EURe and nBTC/USDC.

### Phase 4. Prune integration
- Replace hardcoded prune constants with DB policy.
- Ensure rollup-before-prune ordering.
- Preserve completed successful quote ids across all quote lifecycle tables.
- Add bounded batch and pressure-mode tests.

### Phase 5. Dashboard and health
- Add storage/retention dashboard card or page.
- Wire rollup freshness and table sizes into dashboard bootstrap/WebSocket state.
- Add runtime-health warning fields and alert tests.

### Phase 6. Validation and deploy
- Run targeted retention/rollup/storage tests.
- Run full `npm test`.
- Build dashboard bundle.
- Deploy through repo workflow only.
- Collect live evidence:
  - table sizes before/after at least one prune cycle
  - rollup freshness
  - retained successful evidence count
  - recent quote flow still writing rows
  - all deployments healthy

## Test plan
- Retention classifier tests.
- Successful quote preserve tests.
- Rollup grouping tests:
  - pair
  - direction
  - request kind
  - result code
  - failure category
  - quote-age bucket
  - notional bucket
  - outcome status
- Rollup idempotency tests.
- Policy validation tests.
- Prune blocked when rollup stale.
- Pressure-mode prune tests.
- Old-row compatibility tests.
- Dashboard storage/retention UI tests.
- Runtime-health/alert tests.
- Static config tests proving no new active retention env vars.
- Full `npm test`.
- Dashboard build.

## Validation checklist against the proof
- Successful completed quote evidence remains in detail tables after prune.
- Non-success detail tables stay bounded under live quote flow.
- Rollups continue to answer competitiveness questions after old detail is pruned.
- Dashboard shows retention mode, table sizes, rollup freshness, and prune results.
- No strategy behavior changes.
- Repo workflow deploys the result.

## Known fakes allowed at start of this turn
- Historical non-success detail already pruned cannot be recovered.
- Fee-complete realized PnL is still unavailable.
- Venue-native terminal fill ids remain unavailable.
- Rollups are analytics summaries, not proof of individual trades.