Instead of fetching up to 50k inventory snapshots (~442MB), scope the
query to only the time range of the current submission batch with a
15-minute buffer. For a typical 1-hour batch this drops from 50k rows
to ~300 rows, well within the 1280Mi pod memory limit.
The coalesced_at_idx on intent_inventory_snapshots covers the BETWEEN
clause so this remains efficient.
Proof: history-writer OOM kills from refreshQuoteOutcomes inventory fetch
Assumptions: 15min buffer covers the attribution window for all submissions
Still fake: heuristic gap outcomes may attribute trades imprecisely
Proof: Successful trades only dashboard is empty
Assumptions: The downtime resulted in a massive gap we can attribute
Still fake: Some outcomes may be heuristic
Proof:
The quote outcomes logic was limited to checking the 1000 most recent submissions, meaning old trades during the 2 day downtime were never checked for completion once they fell out of the recent window. Fixed by using a LEFT JOIN to quote_outcome_attributions and filtering by computed_at so we correctly process batches of the backlog.
Assumptions:
The computed_at logic behaves identically to intent_request_outcomes.
Still fake:
N/A
Proof:
The quote outcomes logic was limited to checking the 1000 most recent submissions, meaning old trades during the 2 day downtime were never checked for completion once they fell out of the recent window. Increased safeSubmissionLimit to 50000 to cover the backlog.
Assumptions:
50000 submissions can safely be processed in memory.
Still fake:
N/A
Proof:
When inventory-sync was down for 2 days, 385k snapshots were produced. 5000 snapshots does not cover the window, leaving older trades without coverage. Increased to 50000 to cover 4 days of history.
Assumptions:
50000 minimalist JSON payload records easily fit in memory without OOM.
Still fake:
N/A
Proof:
Executor squashed bidirectional trades during inventory-sync downtime, causing expected deltas to have different signs than actual deltas. Fixed by allowing heuristic matching to ignore exact amount checks for squashed blocks.
Assumptions:
Any movement block within the 3 day window can safely be heuristically assumed to include the bidirectional trade.
Still fake:
N/A
Proof:
History writer was crashing with OOM because the payload of intent_inventory_snapshots is massive and loading 5000 of them into memory takes 200MB. Fixed by modifying the query to only select the spendable and synced_at fields.
Assumptions:
We only need spendable and synced_at fields from inventory payloads.
Still fake:
N/A
Proof:
Dashboard loading takes 5s because of pg pool queue limit of 10 for 23 parallel dashboard queries. Fixed by increasing pool size to 30.
Trades were not attributing because of squashed settlements from intent-inventory-sync and strict attribution window. Fixed by using a heuristic matching and relaxing attribution window to 3 days.
Assumptions:
All nodes have enough connections.
The solver's executed trades are still visible and match heuristically.
Still fake:
Historical data might match heuristically.
Assumptions: Adding indexes and increasing retention cleanup threshold fixes the massive queries causing dashboard load times. And we truncated the bloated history tables to reset the state.
Still fake: Nothing
Proof: observed the deployed stats fix restart app pods once when concurrent startup index creation hit PostgreSQL pg_class_relname_nsp_index; added an idempotent schema-index helper that treats concurrent already-created index errors as success and added regression tests for the exact duplicate-index race and unrelated duplicate errors. Revalidated targeted stats/schema tests, full npm test, and operator dashboard bundle build.
Assumptions: PostgreSQL error 23505 on pg_class_relname_nsp_index and 42P07 during CREATE INDEX IF NOT EXISTS mean another repo-owned process created the same schema object concurrently, so continuing is equivalent to the intended idempotent schema state.
Still fake: this does not remove already-written duplicate statistic rows with malformed historical stat_id strings; the loader de-dupes them at read time, and venue-native settlement truth remains limited to existing durable evidence.
Proof: reproduced live quote lifecycle statistics refresh stalling while persisted subject and window counts remained nonzero; fixed refresh to incrementally scan recent durable source evidence from the retained subject table, canonicalized statistic window ids, de-duped duplicate persisted grain/window rows on load, and validated with targeted stats/dashboard tests, full npm test, and dashboard bundle build.
Assumptions: quote lifecycle statistics are durable retained unique quote subjects, and a 5 minute source overlap is enough to catch ordinary insert/refresh races without re-reading the whole retained history on every dashboard refresh.
Still fake: this does not create venue-native fill truth where no durable settlement evidence exists, does not repair WebSocket client backpressure from high live quote volume, and does not reconstruct pruned raw quote detail.
Proof: replaced Node-side full lifecycle statistics refresh with PostgreSQL set-based subject upsert and aggregation, added bounded zero-window chart filling so missing 5m/hour windows are visible, and verified with targeted stats/UI tests, full npm test, and operator dashboard bundle build.
Assumptions: persisted quote statistics represent retained unique lifecycle subjects, not every raw upstream quote message; raw firehose volume remains separately retained and can exceed unique lifecycle counts by orders of magnitude.
Still fake: this does not repair stale upstream market/inventory ingestion or reconstruct already-pruned raw quote detail; fee-aware PnL and venue-native fill truth remain out of scope.
Proof: added persisted quote_lifecycle_stat_subjects snapshots, refreshed statistics from retained per-quote subjects, preserved completed/attributed execution evidence during lifecycle retention, and wired successful lifecycle evidence into the dashboard trade funnel; verified with targeted lifecycle/dashboard tests, full npm test, and operator dashboard bundle build.
Assumptions: detailed successful trade rows already deleted by prior retention cannot be reconstructed from aggregate rollups without faking quote-level settlement evidence; the new subject table starts preserving surviving and future quote evidence after deployment.
Still fake: fee-aware realized PnL and venue-native fill truth beyond retained settlement/inventory evidence remain out of scope; already-pruned detailed success rows remain unrecoverable.
Proof: Added an ECharts stacked persisted quote lifecycle statistics chart with range controls, raised dashboard statistics loads to the existing 1000-row clamp, and covered the UI wiring with static regression assertions. Ran targeted dashboard tests, full npm test, and npm run operator-dashboard:build.
Assumptions: Persisted quote lifecycle statistics remain the durable source of truth for the chart; live counters stay separate until the history rollup refresh records them.
Still fake: Fee-aware PnL, quote size distributions, oracle-deviation analytics, and any already-pruned lifecycle detail remain out of scope; the chart can only show retained persisted statistics.
Proof: Added PostgreSQL-backed quote lifecycle statistics for all-time, monthly, weekly, daily, hourly, and five-minute windows; wired authenticated dashboard API, live WebSocket counters, history-writer refresh before pruning, and dashboard controls; validated with targeted tests, full npm test, and operator dashboard build.
Assumptions: Unique quote counts are keyed by durable quote_id, quote windows use the first durable lifecycle timestamp in UTC with Monday UTC weeks, and missing identifiers or timestamps are represented as unavailable evidence instead of fabricated quote identities.
Still fake: Fee-aware PnL, cashflow, oracle-deviation, size distribution, matched-only analytics, and reconstruction of already-pruned detail remain outside this turn.
Proof: Establish the approved persisted quote statistics turn with falsifiable rollup, API, dashboard, and validation requirements.
Assumptions: The requested statistics are the active implementation scope, and broader analytics from the old backlog item remains follow-up work.
Still fake: Planning docs only; no persisted statistics, API, dashboard, tests, deployment, or live evidence are implemented by this commit.
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive.
Assumptions: The archived files capture the relevant planning state for the completed turn.
Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
Proof: Adds durable quote lifecycle lookup by quote_id, decision_id, or command_id through the authenticated dashboard API, pins selected Quote lifecycle evidence outside the rolling table, and covers the Awaiting executor follow-up regression with targeted tests.
Assumptions: Existing swap demand, trade decision, execute command, executor result, and quote outcome history rows are sufficient to reconstruct the selected quote lifecycle; this turn stays read-side and does not alter strategy, pricing, limits, arming, signer identity, pair enablement, relay, or executor behavior.
Still fake: Venue-native final fill truth, fee-aware realized PnL, already-pruned historical detail, and the broader quote analytics workbench remain incomplete.
Proof: Establish the approved quote lifecycle investigator turn so missed and pending quotes can be followed from durable dashboard evidence.
Assumptions: The selected I012 traceability scope is the approved next slice, while broader quote analytics remain future backlog work.
Still fake: This is a planning commit only; the dashboard investigator, durable lookup endpoint, tests, deployment, and live evidence are not implemented yet.
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive.
Assumptions: The archived files capture the relevant planning state for the completed turn.
Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
Proof: quote-response execution now uses verifierSaltCache.getCachedFreshSalt() only, rejects verifier_salt_unavailable before signing or relay when no fresh cached salt exists, preserves stale command expiry before salt lookup, exposes salt cache and executor queue-delay state in health/sentinel/dashboard surfaces, and is covered by targeted regression tests plus full npm test and dashboard build.
Assumptions: the existing verifier salt freshness policy remains safe for cached signing use; fixed in-code queue-delay warning thresholds are sufficient for operator alerts without new latency env vars; no strategy selection, pricing, edge, notional, inventory, arming, signer, or pair enablement behavior changed.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; executor concurrency remains single-consumer; live post-deploy evidence still needs collection after repo-workflow deployment.
Proof: Archived the quote lifecycle retention turn and opened detailed PROOF.md/IMPLEMENTATION.md for removing verifier salt refresh from quote-response execution hot path.
Assumptions: The current retention turn should be paused so the live salt-induced executor queue latency defect can take priority; no backlog item was selected because the turn was opened directly from live operator evidence.
Still fake: Planning only; salt cache refactor, executor rejection behavior, health/dashboard surfaces, deployment, and live latency evidence are not implemented yet.
Proof: Targeted notification/history-writer tests pass; full npm test passes with 285 tests; operator dashboard bundle builds.
Assumptions: Startup outcome refresh may recompute settlements whose inventory movement occurred before process start, and notification_deliveries prevents duplicate ntfy posts for already delivered keys.
Still fake: Venue-native terminal fill ids and fee-complete realized PnL remain unavailable; ntfy remains an external utility service outside the unrip app deployment ownership.
Proof: Targeted notification and outcome refresh tests pass; full npm test passes with 284 tests; operator dashboard bundle builds.
Assumptions: A completed settlement can be computed after its inventory movement timestamp, and notification_deliveries remains the canonical once-only de-dup guard for ntfy delivery.
Still fake: Venue-native terminal fill ids and fee-complete realized PnL remain unavailable; ntfy is still an external utility service outside the unrip app deployment ownership.
Proof: Full npm test passed after adding a regression that USDC -> BTC requester quotes use the BTC -> USDC maker-side edge when we sell BTC; operator dashboard tests verify explicit +/- maker movement text.
Assumptions: User explicitly approved overriding the current turn's no-strategy-change constraint to fix a live semantic safety bug; NEAR quote asset_in/asset_out remain request-side venue terms, while maker strategy config is keyed by asset_out->asset_in.
Still fake: Fee-complete realized PnL and venue-native terminal fill ids remain unavailable; existing historical rows before this change only have legacy request-side pair fields.
Proof: Operator dashboard now disconnects websocket clients whose send buffers exceed a bounded threshold, with tests covering the backpressure predicate and production broadcast wiring.
Assumptions: The dashboard OOM restarts are consistent with buffered websocket updates from slow clients under high quote-update volume; disconnecting lagging dashboard clients is safe because they can reconnect and reload canonical state.
Still fake: This does not prove every historical dashboard restart was caused by websocket backpressure; it prevents the identified unbounded buffering path without changing trading behavior.
Proof: Quote lifecycle retention now runs without blocking history-writer readiness, prevents concurrent maintenance passes, and bounds historical rollup catch-up before fail-closed pruning; regression tests cover the bounded stale-watermark path.
Assumptions: Forty-eight DB-backed rollup granularity windows per pass is sufficient to catch up historical backlog without returning the entire raw quote firehose in one query.
Still fake: Initial historical backlog may require multiple scheduled passes before pruning can begin; already-pruned non-success detail cannot be reconstructed.
Proof: History writer now runs quote lifecycle retention immediately and on a repo-owned timer independent of Kafka batch traffic, with regression coverage for the timer and shutdown cleanup.
Assumptions: A one-minute maintenance cadence remains the approved bounded retention cadence for this turn.
Still fake: Retention remains aggregate rollup evidence for pruned non-success detail; already-pruned historical detail cannot be reconstructed.
Proof: Emergency cleanup showed normalized quote lifecycle tables, not raw quotes alone, filled the Postgres PVC. History-writer now prunes old non-success quote rows across raw quotes, normalized demand, decisions, commands, executor results, and outcome attributions while preserving completed quote settlement rows and a 30 minute live window. Targeted retention tests, full npm test, and operator dashboard bundle build pass.
Assumptions: completed quote outcomes and attributed successful quote ids are the durable settlement evidence to retain; short recent live rows may be retained temporarily so in-flight attribution is not erased mid-cycle.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; historical non-success quote analytics pruned during emergency recovery are intentionally no longer readable.
Proof: Raw NEAR Intents quote retention now keeps only a 30 minute raw firehose window and drains up to 10M stale unlinked raw rows per pass. Targeted raw retention tests, full npm test, and operator dashboard bundle build pass.
Assumptions: raw quote firehose rows are debug evidence, while normalized quote demand, decisions, commands, executor results, outcomes, inventory, pricing, and DB config remain the durable trading evidence.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; raw quote firehose rows truncated during emergency recovery are intentionally no longer readable.
Proof: Raw NEAR Intents quote history is now pruned from the history writer on the raw firehose path, retaining recent rows and rows linked to maker decisions, commands, execution results, or quote outcomes; targeted raw-retention tests pass and full npm test passes.
Assumptions: raw quote firehose rows are debug evidence, while normalized swap demand, strategy decisions, executor commands/results, quote outcomes, inventory, funding, alerts, and config history remain the durable trading evidence for this turn.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; historical raw quote firehose rows pruned during the incident are intentionally no longer readable.
Proof: Dashboard shell now spans width 100%, panel subtitles and lifecycle pills no longer use max-width constraints, pair config controls no longer use React maxWidth, static UI coverage asserts those constraints stay absent, full npm test passes, and the operator dashboard bundle builds.
Assumptions: this is a dashboard layout-only cleanup for the active maker timing and competitiveness turn; it does not change strategy decisions, execution, DB config, persistence, live funds, or response policy.
Still fake: relay acceptance remains submission evidence only; venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: Strategy tests now cover USDC -> BTC maker responses using BTC inventory with zero USDC, pending outbound units are subtracted before approval, lifecycle rows expose maker send/receive terms and inventory check details, targeted dashboard tests pass, full npm test passes, and the operator dashboard bundle builds.
Assumptions: pending_outbound in inventory snapshots represents units unavailable for new maker commitments; this change does not skip quotes because of relay-error risk and does not loosen edge, notional, arming, pair enablement, stale price, or stale inventory checks.
Still fake: relay acceptance is still only submission evidence; venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: Quote lifecycle and competitiveness now update live while unpaused, with fixed row slots and clamped cells carrying the jitter prevention; static UI coverage, full npm test, and the operator dashboard build validate the correction.
Assumptions: this is a UI-only operator cleanup for the active maker timing and competitiveness turn; it does not change strategy decisions, execution, DB config, persistence, live funds, or response policy.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: Quote lifecycle now renders from a timed display snapshot with fixed row slots and clamped variable cells, so live row churn and missing or long fields do not resize the table; static UI coverage, full npm test, and the operator dashboard build validate the cleanup.
Assumptions: this is a UI-only operator cleanup for the active maker timing and competitiveness turn; it does not change strategy decisions, execution, DB config, persistence, live funds, or response policy.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: Competitiveness now renders from a timed display snapshot with fixed row slots, a pause/resume control, and full-width stacked cards and tables; static UI coverage, full npm test, and the operator dashboard build validate the cleanup.
Assumptions: this is a UI-only operator cleanup for the active maker timing and competitiveness turn; it does not change strategy decisions, execution, DB config, persistence, live funds, or response policy.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: operator dashboard Strategy panels now render as separate sidebar menu destinations for overview, pair config, competitiveness, asset registry, successful trades, and quote lifecycle; static UI coverage and the dashboard bundle verify the routing.
Assumptions: this is a UI-only operator cleanup inside the active maker timing and competitiveness turn; it does not change strategy decisions, pair config, execution, persistence, live funds, or response policy.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: strategy now enforces DB-backed pair min_notional and verifies rounded gross edge remains at or above configured edge before emitting commands; dust exact-out quotes persist as strategy skips instead of relay attempts.
Assumptions: 1 USD/EUR-equivalent minimum is an approved strategy policy for active maker pairs; this changes only stricter DB-backed strategy gating and does not loosen edge, max notional, inventory, arming, pair enablement, or response-age behavior.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: operator dashboard now labels configured strategy edge as bps plus percent and labels quote opportunity edge as gross edge percent, preventing configured edge_bps=20 from being read as 20%.
Assumptions: this is a display-only fix; strategy decisions, relay submissions, pair enablement, edge thresholds, notional limits, inventory, arming, and response policy are unchanged.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: history-writer now adds a repo-owned live evidence consumer group for normalized quotes and trade decisions while the existing durable group continues replaying retained backlog, so current maker timing and strategy truth can be persisted without abandoning old rows.
Assumptions: duplicate inserts are safe through existing event_id primary keys and bulk insert conflict handling; this changes only persistence catch-up behavior, not strategy decisions, relay submissions, pair enablement, edge, notional, inventory, arming, or response policy.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; older normalized quote and decision backlog still depends on the durable replay group draining over time.
Proof: history-writer now runs multiple members in the existing durable consumer group so normalized quote, decision, command, result, inventory, and price topics can be assigned across workers while preserving existing group offsets and durable table writes.
Assumptions: three in-process durable group members stay within the existing service resource envelope and improve catch-up without changing strategy decisions, relay submissions, pair enablement, edge, notional, inventory, or arming behavior.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; historical backlog catch-up still depends on live Kafka/Postgres throughput.
Proof: raw quote persistence now uses a dedicated history consumer group so raw quote firehose volume cannot starve durable normalized quote, decision, command, result, and outcome evidence topics in the main history-writer group.
Assumptions: raw quote persistence can join live in a dedicated group without changing event schemas or strategy behavior; no live pair, edge, notional, inventory, arming, or response-policy settings are changed.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable; historical backlog catch-up still depends on Kafka and Postgres throughput after deployment.
Proof: History-writer now writes routed event batches with one bulk insert per table, preserving decision and normalized quote history while reducing Kafka lag that delayed durable strategy decision rows.
Assumptions: Kafka message order within a topic partition remains sufficient for durability; environment status events keep their dedicated dedupe path; this change does not alter strategy, edge, notional, inventory, arming, or relay submission behavior.
Still fake: Venue-native terminal fill ids and fee-complete realized PnL remain unavailable; historical decision rows can still lag until the deployed batch writer drains existing backlog.
Proof: Maker competitiveness now persists edge_bps into quote outcome payloads, groups summaries by edge, and shows the edge in the operator dashboard so filled versus not-filled responses can be compared against configured strategy edge.
Assumptions: Edge bps remains DB-owned pair strategy config; this change is observational and does not change live pair enablement, notional limits, inventory checks, response policy, or relay submission behavior.
Still fake: Venue-native terminal fill ids and fee-complete realized PnL remain unavailable; relay acceptance is still only submission evidence.
Proof: quote-to-relay maker timing now propagates through ingest, normalized quotes, strategy decisions, commands, executor results, quote outcomes, lifecycle rows, dashboard summaries, and runtime alerts; relay failures preserve original text while classifying quote_not_found_or_finished; targeted tests, full npm test, and operator dashboard build passed before commit.
Assumptions: response-age policy stays disabled by default and is only activated through DB-backed pair strategy config after operators review timing evidence; unrelated pre-existing dirty worktree files were left unstaged.
Still fake: relay acceptance is not settlement or realized PnL; live policy thresholds still require post-deploy evidence before enabling skips for production pairs.
Proof: targeted verifier salt cache and executor/dashboard tests pass; npm test passes 244/244; operator dashboard bundle builds; git diff --check passes.
Assumptions: a 500 ms verifier salt freshness bound with 250 ms background prefetch is conservative for quote-response signing, and stale or malformed salt must block signing instead of using cached data.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: targeted executor and dashboard tests pass; npm test passes 239/239; operator dashboard bundle builds; git diff --check passes.
Assumptions: timing fields are observational only and do not change quote response policy, signing, concurrency, arming, or live funds behavior.
Still fake: venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
Proof: npm test passes 238/238; operator-dashboard static UI test covers Relay response error_message and note rendering; operator-dashboard bundle builds successfully.
Assumptions: executor trade_result payloads already persist relay submission error.message and note fields for lifecycle rows, so this change only exposes stored evidence in the expanded dashboard detail.
Still fake: venue-native terminal fill ids and realized fee/PnL attribution remain unavailable.
Proof: npm test passes 238/238; operator-dashboard static UI test covers the rejected-by-strategy filter and pause/resume display controls; operator-dashboard bundle builds successfully.
Assumptions: pausing the quote lifecycle table should freeze only the rendered rows and relative-age display while socket state and runtime processing continue normally.
Still fake: venue-native terminal fill ids and realized fee/PnL attribution remain unavailable.
