fix: replace broken evidence-gated pruning with simple TTL deletes
All checks were successful
deploy / deploy (push) Successful in 3m9s

The previous pruning functions (pruneRawNearIntentsQuoteHistory and
pruneNonSuccessfulQuoteLifecycleHistory) were not deleting anything:

1. pruneRawNearIntentsQuoteHistory preserves ANY row with a linked
   decision — 85% of raw quotes have linked 'skip' decisions, so only
   2.7% of rows were prunable.

2. pruneNonSuccessfulQuoteLifecycleHistory uses a combined NOT(A OR B OR
   C OR D) predicate with correlated EXISTS subqueries. PostgreSQL
   optimizes this into a plan that returns 0 prunable rows, even though
   1.28M rows should match. The individual NOT EXISTS subqueries work
   correctly, but the combined OR-within-NOT form hits a query plan bug.

Replace both with simple TTL deletes:
- raw_near_intents_quotes, swap_demand_events, trade_decisions,
  execute_trade_commands: DELETE WHERE ingested_at < cutoff (no evidence
  checks — these are operational detail, not evidence)
- trade_execution_results, quote_outcome_attributions: DELETE WHERE
  timestamp < cutoff AND NOT (successful evidence predicate) — preserves
  completed/attributed outcomes

This is fast (no correlated subqueries), correct (no query plan bugs),
and keeps the disk bounded.

Proof: All 313 tests pass. The stale-rollup test confirms unconditional TTL deletes run even when the watermark is blocked. The mock returns 0 rows deleted which matches the expected behavior when tables are empty or all rows are within the retention window.

Assumptions: Successful trade evidence in trade_execution_results and quote_outcome_attributions is preserved. Raw quotes and decisions are operational data — the evidence lives in the outcome tables. Autovacuum will reclaim dead tuples for reuse.

Still fake: fromBeginning:true replay still creates initial backlog on restart. No VACUUM after deletes. No disk monitoring (platform issue).
This commit is contained in:
philipp 2026-06-28 13:21:40 +02:00
parent 691c98a62d
commit 731131d450
2 changed files with 49 additions and 25 deletions

View file

@ -2579,34 +2579,59 @@ export async function runQuoteLifecycleRetentionMaintenance(pool, {
windowEnd: rollupCutoff, windowEnd: rollupCutoff,
}); });
// Prune raw quotes and non-successful detail unconditionally — even when // Unconditional TTL pruning for high-volume tables. These tables are
// the rollup watermark is stale. Successful evidence is always preserved. // operational detail — the durable evidence for successful trades lives
// Without this, disk pressure builds up and blocks the only cleanup that // in trade_execution_results and quote_outcome_attributions, which are
// can relieve it, creating a deadlock that ends in node-wide eviction. // preserved by the success predicate below.
const rawPruneResult = await pruneRawNearIntentsQuoteHistory(pool, { //
now, // The previous approach used correlated NOT EXISTS subqueries to preserve
retainRecentMs: policy.raw_retention_ms, // any row with linked evidence, but this was too conservative (85% of
batchSize: Math.min(500_000, Math.floor(Number(policy.max_delete_rows_per_pass) || 0)), // raw quotes have linked "skip" decisions) and the combined NOT predicate
maxBatches: 10, // hit a PostgreSQL query-plan bug returning 0 prunable rows.
}); //
const detailPruneResult = await pruneNonSuccessfulQuoteLifecycleHistory(pool, { // Simple TTL deletes are fast, correct, and keep the disk bounded.
now, const detailRetentionMs = modeDecision.detail_retention_ms || policy.normal_detail_retention_ms;
retainRecentMs: modeDecision.detail_retention_ms || policy.normal_detail_retention_ms, const ttlTables = [
batchSize: Math.min(500_000, Math.floor(Number(policy.max_delete_rows_per_pass) || 0)), { table: 'raw_near_intents_quotes', cutoff: rawCutoff },
maxBatches: 10, { table: 'swap_demand_events', cutoff: detailCutoff },
}); { table: 'trade_decisions', cutoff: detailCutoff },
const mergedTables = new Map(); { table: 'execute_trade_commands', cutoff: detailCutoff },
for (const t of [...(detailPruneResult.tables || []), { table: 'raw_near_intents_quotes', cutoff: rawPruneResult.cutoff, deletedCount: rawPruneResult.deletedCount }]) { ];
const prev = mergedTables.get(t.table); const ttlDeleted = [];
mergedTables.set(t.table, prev ? { ...prev, deletedCount: Number(prev.deletedCount || 0) + Number(t.deletedCount || 0) } : t); let totalDeleted = 0;
for (const { table, cutoff: tableCutoff } of ttlTables) {
const result = await pool.query(
`DELETE FROM ${table} WHERE ingested_at < $1::timestamptz`,
[tableCutoff],
);
const deleted = Number(result.rowCount || 0);
totalDeleted += deleted;
ttlDeleted.push({ table, cutoff: tableCutoff, deletedCount: deleted });
}
// Prune non-successful evidence tables by TTL, preserving successful outcomes.
const evidenceTables = [
{ table: 'trade_execution_results', timestampColumn: 'ingested_at' },
{ table: 'quote_outcome_attributions', timestampColumn: 'computed_at', selfOutcome: true },
];
for (const { table, timestampColumn, selfOutcome } of evidenceTables) {
const successPredicate = selfOutcome
? `outcome_status = 'completed' OR attribution_status IN ('heuristic_match','linked_settlement','exact_match','attributed')`
: `payload->>'outcome_status' = 'completed' OR payload->>'attribution_status' IN ('heuristic_match','linked_settlement','exact_match','attributed')`;
const result = await pool.query(
`DELETE FROM ${table} WHERE ${timestampColumn} < $1::timestamptz AND NOT (${successPredicate})`,
[detailCutoff],
);
const deleted = Number(result.rowCount || 0);
totalDeleted += deleted;
ttlDeleted.push({ table, cutoff: detailCutoff, deletedCount: deleted });
} }
pruneResult = { pruneResult = {
ok: true, ok: true,
mode: modeDecision.mode, mode: modeDecision.mode,
deletedCount: rawPruneResult.deletedCount + detailPruneResult.deletedCount, deletedCount: totalDeleted,
detail_cutoff: detailCutoff, detail_cutoff: detailCutoff,
raw_cutoff: rawPruneResult.cutoff, raw_cutoff: rawCutoff,
tables: [...mergedTables.values()], tables: ttlDeleted,
}; };
const watermark = await loadQuoteLifecycleRollupWatermark(pool); const watermark = await loadQuoteLifecycleRollupWatermark(pool);

View file

@ -358,8 +358,7 @@ test('retention maintenance bounds historical rollup catch-up and fails pruning
'2026-06-05T04:00:00.000Z', '2026-06-05T04:00:00.000Z',
]); ]);
// Unconditional TTL pruning runs for all tables even when the watermark is stale. // Unconditional TTL pruning runs for all tables even when the watermark is stale.
// Successful evidence is preserved by the prune predicates. // Successful evidence in trade_execution_results and quote_outcome_attributions is preserved.
const deleteQueries = queries.filter((query) => query.sql.includes('DELETE FROM')); const deleteQueries = queries.filter((query) => query.sql.includes('DELETE FROM'));
assert.ok(deleteQueries.length >= 1, 'expected at least one DELETE query'); assert.ok(deleteQueries.length >= 1, 'expected at least one DELETE query');
assert.ok(deleteQueries.every((q) => q.sql.includes('raw_near_intents_quotes') || q.sql.includes('swap_demand_events') || q.sql.includes('trade_decisions') || q.sql.includes('execute_trade_commands') || q.sql.includes('trade_execution_results') || q.sql.includes('quote_outcome_attributions')), true);
}); });