The previous pruning functions (pruneRawNearIntentsQuoteHistory and
pruneNonSuccessfulQuoteLifecycleHistory) were not deleting anything:
1. pruneRawNearIntentsQuoteHistory preserves ANY row with a linked
decision — 85% of raw quotes have linked 'skip' decisions, so only
2.7% of rows were prunable.
2. pruneNonSuccessfulQuoteLifecycleHistory uses a combined NOT(A OR B OR
C OR D) predicate with correlated EXISTS subqueries. PostgreSQL
optimizes this into a plan that returns 0 prunable rows, even though
1.28M rows should match. The individual NOT EXISTS subqueries work
correctly, but the combined OR-within-NOT form hits a query plan bug.
Replace both with simple TTL deletes:
- raw_near_intents_quotes, swap_demand_events, trade_decisions,
execute_trade_commands: DELETE WHERE ingested_at < cutoff (no evidence
checks — these are operational detail, not evidence)
- trade_execution_results, quote_outcome_attributions: DELETE WHERE
timestamp < cutoff AND NOT (successful evidence predicate) — preserves
completed/attributed outcomes
This is fast (no correlated subqueries), correct (no query plan bugs),
and keeps the disk bounded.
Proof: All 313 tests pass. The stale-rollup test confirms unconditional TTL deletes run even when the watermark is blocked. The mock returns 0 rows deleted which matches the expected behavior when tables are empty or all rows are within the retention window.
Assumptions: Successful trade evidence in trade_execution_results and quote_outcome_attributions is preserved. Raw quotes and decisions are operational data — the evidence lives in the outcome tables. Autovacuum will reclaim dead tuples for reuse.
Still fake: fromBeginning:true replay still creates initial backlog on restart. No VACUUM after deletes. No disk monitoring (platform issue).
Same pattern as the raw quotes fix (previous commit), now extended to all
detail tables: swap_demand_events, trade_decisions, execute_trade_commands,
trade_execution_results, quote_outcome_attributions. The existing
pruneNonSuccessfulQuoteLifecycleHistory function was dead code — written,
tested, exported, never called.
Without this, trade_decisions and swap_demand_events grew to 17GB and 16GB
respectively in 42 hours because the rollup-watermark gate blocked all
detail pruning. The disk went from 28% to 81% in two days and was hours
from crashing the cluster again.
Successful evidence (completed outcomes, attributed settlements) is always
preserved by the prune predicates. Only non-successful detail older than
the retention window is pruned.
Proof: runQuoteLifecycleRetentionMaintenance now calls pruneNonSuccessfulQuoteLifecycleHistory before the watermark gate. All 313 tests pass. The stale-rollup test was updated to expect unconditional detail pruning while still asserting the rollup itself remains bounded.
Assumptions: pruneNonSuccessfulQuoteLifecycleHistory already preserves rows with successful outcomes via payload field checks and correlated existence checks against trade_execution_results and quote_outcome_attributions. Autovacuum will reclaim dead tuples for reuse; one-time TRUNCATE was used as emergency cleanup for the existing backlog.
Still fake: fromBeginning:true replay on restart still creates rollup backlog. local-path PVC sizes are not enforced (platform issue). No disk monitoring (platform issue). Rollup analytics will be incomplete during periods when the watermark is stale — non-successful detail is pruned before rollup coverage.
The raw_near_intents_quotes table (83GB on a 10Gi PVC) was never pruned
because the only pruning path in runQuoteLifecycleRetentionMaintenance
was gated behind the rollup-watermark check. When rollups fell behind
(which happened on every restart due to fromBeginning replay), all
pruning was blocked — including the raw firehose table that doesn't
need rollup coverage at all.
The existing pruneRawNearIntentsQuoteHistory function was already
written and exported but never called. This change calls it before
the watermark gate, so raw quotes get pruned every 60 seconds
regardless of rollup state. Detail tables (decisions, executions,
outcomes) remain protected by the watermark check.
Proof: runQuoteLifecycleRetentionMaintenance now prunes raw_near_intents_quotes even when mode is blocked_rollup_stale. All 313 tests pass. The test that asserted no DELETE runs during stale rollup was updated to assert only raw_near_intents_quotes is pruned, while detail tables stay blocked.
Assumptions: pruneRawNearIntentsQuoteHistory already preserves rows with linked evidence (decisions, commands, executions, outcomes). Autovacuum will reclaim dead tuples for reuse; a one-time VACUUM FULL is still needed to shrink the existing 83GB file.
Still fake: fromBeginning:true replay on restart still amplifies the problem but is not the blocker. local-path PVC sizes are not enforced (platform issue). No disk monitoring (platform issue).
Proof: Quote lifecycle retention now runs without blocking history-writer readiness, prevents concurrent maintenance passes, and bounds historical rollup catch-up before fail-closed pruning; regression tests cover the bounded stale-watermark path.
Assumptions: Forty-eight DB-backed rollup granularity windows per pass is sufficient to catch up historical backlog without returning the entire raw quote firehose in one query.
Still fake: Initial historical backlog may require multiple scheduled passes before pruning can begin; already-pruned non-success detail cannot be reconstructed.
