Compare commits

..

3 commits

Author SHA1 Message Date
philipp
cb87910c95 Implement quote lifecycle investigator
All checks were successful
deploy / deploy (push) Successful in 57s
Proof: Adds durable quote lifecycle lookup by quote_id, decision_id, or command_id through the authenticated dashboard API, pins selected Quote lifecycle evidence outside the rolling table, and covers the Awaiting executor follow-up regression with targeted tests.

Assumptions: Existing swap demand, trade decision, execute command, executor result, and quote outcome history rows are sufficient to reconstruct the selected quote lifecycle; this turn stays read-side and does not alter strategy, pricing, limits, arming, signer identity, pair enablement, relay, or executor behavior.

Still fake: Venue-native final fill truth, fee-aware realized PnL, already-pruned historical detail, and the broader quote analytics workbench remain incomplete.
2026-06-12 19:28:38 +02:00
philipp
0902cb2c93 Open implementation turn: Quote lifecycle investigator
Proof: Establish the approved quote lifecycle investigator turn so missed and pending quotes can be followed from durable dashboard evidence.

Assumptions: The selected I012 traceability scope is the approved next slice, while broader quote analytics remain future backlog work.

Still fake: This is a planning commit only; the dashboard investigator, durable lookup endpoint, tests, deployment, and live evidence are not implemented yet.
2026-06-12 19:03:13 +02:00
philipp
cf90597678 Archive implementation turn: Verifier salt hot-path removal and executor queue guardrails
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive.
Assumptions: The archived files capture the relevant planning state for the completed turn.
Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
2026-06-12 18:45:53 +02:00
15 changed files with 1950 additions and 360 deletions

View file

@ -22,6 +22,7 @@ Legacy note:
- 2026-06-06: `maker-response-latency-and-quote-competitiveness` closed with status `passed`. Durable maker timing, relay failure classification, pair-native competitiveness surfaces, dashboard cleanup, and emergency quote-history retention fixes were implemented, validated, and deployed; long-term storage shape is opened as a separate turn. - 2026-06-06: `maker-response-latency-and-quote-competitiveness` closed with status `passed`. Durable maker timing, relay failure classification, pair-native competitiveness surfaces, dashboard cleanup, and emergency quote-history retention fixes were implemented, validated, and deployed; long-term storage shape is opened as a separate turn.
- 2026-06-12: `quote-lifecycle-retention-and-analytics-rollups` closed with status `paused`. Paused after retention, dashboard, semantic, and notification fixes were deployed; the next urgent proof is removing verifier salt refresh from the quote execution hot path. - 2026-06-12: `quote-lifecycle-retention-and-analytics-rollups` closed with status `paused`. Paused after retention, dashboard, semantic, and notification fixes were deployed; the next urgent proof is removing verifier salt refresh from the quote execution hot path.
- 2026-06-12: `verifier-salt-hot-path-removal-and-executor-queue-guardrails` closed with status `passed`. Verifier current_salt RPC was removed from the quote-response hot path, cached salt failures now reject before signing or relay, queue delay and salt health are visible and alertable, and live post-deploy evidence showed cache or unavailable salt results with no new-runtime refresh rows.
## Research Turns ## Research Turns
## Planning Events ## Planning Events
@ -43,3 +44,4 @@ Legacy note:
- 2026-05-18: opened implementation turn `maker-response-latency-and-quote-competitiveness` from backlog items none. - 2026-05-18: opened implementation turn `maker-response-latency-and-quote-competitiveness` from backlog items none.
- 2026-06-06: opened implementation turn `quote-lifecycle-retention-and-analytics-rollups` from backlog items none. - 2026-06-06: opened implementation turn `quote-lifecycle-retention-and-analytics-rollups` from backlog items none.
- 2026-06-12: opened implementation turn `verifier-salt-hot-path-removal-and-executor-queue-guardrails` from backlog items none. - 2026-06-12: opened implementation turn `verifier-salt-hot-path-removal-and-executor-queue-guardrails` from backlog items none.
- 2026-06-12: opened implementation turn `quote-lifecycle-investigator-and-missed-quote-follow-up` from backlog items I012; remaining broad analytics-workbench scope was re-added as I017.

View file

@ -17,10 +17,10 @@ Rules:
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory. - [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state. - [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility. - [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
- [I012] Quotes and execution analytics workbench: live quote tape, quote count and volume windows, size-bucket distributions, oracle-deviation views, per-quote decision and execution trace, and matched-only filters.
- [I013] Benchmark and PnL analytics: compare trade PnL versus mark-to-market, all-BTC hold, all-EURe hold, passive 50/50 hold, and hindsight trade quality against later benchmarks. - [I013] Benchmark and PnL analytics: compare trade PnL versus mark-to-market, all-BTC hold, all-EURe hold, passive 50/50 hold, and hindsight trade quality against later benchmarks.
- [I016] (soon) Treasury cashflow and fee ledger: durably record deposits, withdrawals, bridge or network costs, and other non-trade cashflows so dashboard profit can move from mark-to-market to true net PnL. tags=pnl,fees,treasury - [I016] (soon) Treasury cashflow and fee ledger: durably record deposits, withdrawals, bridge or network costs, and other non-trade cashflows so dashboard profit can move from mark-to-market to true net PnL. tags=pnl,fees,treasury
- [I017] (soon) Remaining quote analytics workbench beyond lifecycle investigation: quote count and volume windows, size-bucket distributions, oracle-deviation views, matched-only filters, and broader execution analytics. tags=analytics,dashboard,quotes
## Research Candidates ## Research Candidates
- [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair. - [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair.
- [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold. - [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold.

View file

@ -1,235 +1,190 @@
# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails # Implementation Turn: Quote lifecycle investigator and missed quote follow-up
Status: open Status: open
Opened: 2026-06-12 Opened: 2026-06-12
## Goal ## Goal
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators. Improve the dashboard Quote lifecycle view so operators can pin, search, and follow missed or pending quotes from durable evidence even after live rows move on.
## Selected backlog items ## Selected Backlog Items
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12. - [I012] Quotes and execution analytics workbench: live quote tape, quote count and volume windows, size-bucket distributions, oracle-deviation views, per-quote decision and execution trace, and matched-only filters.
## Design rules This turn intentionally implements only the per-quote lifecycle investigation and missed-quote follow-up part of I012. Remaining analytics-workbench work has been re-added to backlog as follow-on scope.
- This turn is execution latency and visibility only.
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement. ## Constraints
- Do not skip quotes because relay errors are likely. - Preserve trading behavior: no strategy, pricing, threshold, notional, inventory, arming, signer, pair enablement, quote skip, or relay policy changes.
- Do not submit a quote response if signing prerequisites are unavailable.
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars. - Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
- Keep old executor result rows readable. - Do not introduce a new storage system or retention policy.
- Deploy only through repo workflow. - Preserve existing lifecycle and executor result rows as readable.
- Use repo workflow for deployment only; no manual kubectl rollout or image patching.
- Add regression tests for every bug fix or behavior contract introduced here.
## Problem statement ## Current Shape To Inspect
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing. - `src/operator-dashboard/static/pages/StrategyPage.jsx`
- `QuoteLifecycleSection`
- `QuoteLifecycleTable`
- `LifecycleDetails`
- pause/filter/expanded-row behavior
- `src/operator-dashboard/static/state/dashboardReducer.js`
- `quote_lifecycle.updated` handling
- `src/core/operator-dashboard.mjs`
- `deriveQuoteLifecycleRows`
- `buildLiveQuoteLifecycleRows`
- `buildStrategySummary`
- lifecycle row normalization/enrichment
- `src/lib/postgres.mjs`
- recent quote, decision, command, execution result, and outcome loaders
- `src/apps/operator-dashboard.mjs`
- bootstrap data loading
- REST API routes
- WebSocket update path
- Existing dashboard/static tests under `test/`.
Live evidence showed this can block for seconds: ## Implementation Plan
- `ae824f64-ddfd-4310-a44f-69c401fc8ad7` ### 1. Add a durable lifecycle lookup contract
- `current_salt_source=refresh` - Add a backend loader that can find lifecycle evidence by one identifier:
- `current_salt_ms=7481.458` - `quote_id`
- executor received the command quickly, then blocked inside salt refresh - `decision_id`
- following commands waited in the single executor consumer queue: - `command_id`
- `command_to_executor_ms` around `7.5s` - Reuse existing durable tables and normalizers:
- some later rows around `10-12s` - `swap_demand_events`
- most failed with `quote_not_found_or_finished` - `trade_decisions`
- `execute_trade_commands`
- `trade_execution_results`
- quote outcomes table
- Return the same normalized lifecycle row shape produced by `deriveQuoteLifecycleRows`, enriched for UI in the same way as recent rows.
- Include lookup metadata:
- requested identifier
- matched identifier type
- found/not found
- lookup timestamp
- latest stage timestamp
- If no row is found, return a structured not-found response instead of a dashboard error.
- Make missing old fields render as `null`/`Unavailable`; do not require new timing fields to exist.
External verifier latency must not sit on the per-quote execution critical path. ### 2. Expose a dashboard API endpoint
- Add an authenticated operator-dashboard route for lifecycle lookup, for example:
- `GET /api/strategy/quote-lifecycle/:identifier`
- The endpoint should:
- validate a non-empty identifier
- call the durable lookup loader
- return `{ ok, lifecycle_row, lookup }`
- return a 404-style payload for not found without crashing the dashboard
- Do not expose secrets or raw relay payloads beyond what existing lifecycle rows already show.
## Backend changes ### 3. Improve frontend state and interaction model
- Replace the current "pause solves investigation" behavior with a selected quote investigator:
- row action: inspect or pin quote
- selected row highlight
- stable investigator panel above the table or directly below controls
- selected quote remains visible even when filters change or live rows move
- Keep pause as table-only:
- paused table keeps row order and row contents stable
- selected investigator can still refresh from durable lookup
- resume table applies latest rolling rows without clearing the selected quote
- Add identifier lookup:
- search input for quote id, decision id, or command id
- loading, found, not-found, and error states
- copyable identifiers remain available
- Add problem-state filters without changing lifecycle data:
- all
- awaiting executor
- approved/no command
- blocked before submit
- submission failed
- quote not found or finished
- strategy rejected
- submitted/awaiting outcome
- Ensure filters do not hide or clear the selected investigator.
### 1. Inspect current salt and executor paths ### 4. Make `Awaiting executor` explain the next step
- Inspect: - In the investigator panel, show a concise stage status:
- `src/core/verifier-salt-cache.mjs` - quote observed
- `src/apps/trade-executor.mjs` - strategy decision
- executor command expiry helpers - command recorded or missing
- executor timing payload construction - executor result recorded or missing
- runtime health and service snapshot summaries - relay/outcome status when present
- ops sentinel alert generation - For `command_emitted` / `Awaiting executor` rows:
- operator dashboard service/system views - show command age using the current clock
- existing verifier salt cache and executor tests - show the command id and command timestamp
- Identify every call path that can refresh verifier salt. - show "waiting for executor result" only when no result is found
- Confirm which call paths are quote-response hot path versus background or operator-triggered work. - if a later lookup finds an executor result, show the terminal status instead of stale awaiting text
- For relay failures, make `quote_not_found_or_finished` visible as a first-class reason.
### 2. Define salt cache contract ### 5. Keep live updates and lookup refresh coherent
- Add or refine a cache API with separate responsibilities: - When `quote_lifecycle.updated` arrives:
- background refresh may perform network I/O - update the rolling table as it does today
- hot-path read must never perform network I/O - if the selected quote appears in the live payload, refresh the selected panel from that row immediately
- Candidate API: - if the selected quote is absent from the live payload, schedule or expose a durable lookup refresh for the selected identifier
- `start()` starts periodic refresh and immediate refresh attempt - Avoid unbounded request loops:
- `refreshNow()` performs explicit refresh with in-flight de-duplication - debounce lookup refreshes client-side
- `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O - do not add new env vars for refresh timing
- `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error - keep manual refresh available
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API. - Show whether selected evidence came from:
- Validate malformed salt before it can enter the cache. - current live rows
- durable lookup
- not found
- lookup error
### 3. Background refresh behavior ### 6. Preserve layout quality
- Ensure startup kicks off refresh without making the executor command handler wait on it. - Keep the lifecycle table full-width and scan-friendly.
- Refresh periodically before the salt becomes stale. - Avoid large marketing-style or decorative redesign.
- If refresh fails: - Keep row heights stable with existing fixed row slots.
- keep a still-fresh cached salt usable until its age limit expires - Ensure long quote ids and reason text truncate or wrap intentionally.
- expose last error and stale/unavailable state - Do not nest cards inside cards; use a panel/detail layout consistent with the existing dashboard.
- do not block command handling - Keep old rows with missing fields readable on desktop and mobile widths.
- Record:
- `last_refresh_started_at`
- `last_refresh_completed_at`
- `last_refresh_duration_ms`
- `last_refresh_error`
- `refresh_in_flight`
- `salt_age_ms`
- `salt_fresh`
### 4. Trade-executor hot path ### 7. Tests
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read. - Add or update core tests for lifecycle lookup and row derivation:
- If no fresh cached salt is available: - command-only row classifies as `Awaiting executor`
- publish a durable trade execution result - later executor result for the same quote updates the lifecycle state
- use status `rejected` - old rows missing timing fields normalize safely
- use explicit result code such as `verifier_salt_unavailable` - Add backend API tests or static route tests:
- include salt cache state summary in the result payload - lookup by quote id
- do not call `buildQuoteResponseSubmission` - lookup by decision id
- do not call the relay - lookup by command id
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`. - not-found response
- Keep idempotency and executor state-store semantics intact. - Add frontend/static tests proving:
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable. - Quote lifecycle has a selected/pinned investigator
- pause language is table-specific
- identifier lookup controls exist
- problem-state filters include awaiting executor and quote-not-found/finished
- selected investigation is not represented only by expanded table rows
- Add a regression test for the described bug:
- start with a selected `Awaiting executor` quote
- update the displayed recent list so that quote is absent
- prove the selected quote detail remains available or can be refreshed from durable lookup
### 5. Queue delay guardrails ### 8. Validation Commands
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`. - Run targeted tests first, likely:
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics. - `node --test test/operator-dashboard*.test.mjs test/*lifecycle*.test.mjs`
- Add thresholded runtime health fields: - add narrower filenames as tests are introduced
- recent max command-to-executor - Run full:
- p90 or bucketed queue delay if practical - `npm test`
- latest command queue delay warning reason - Build dashboard bundle:
- Do not use queue delay to change strategy selection or pricing in this turn. - `npm run operator-dashboard:build`
### 6. Runtime health and alerts ### 9. Commit, deploy, and evidence
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status. - Commit implementation with required body:
- Add ops-sentinel warning alerts for: - `Proof: ...`
- verifier salt stale or unavailable - `Assumptions: ...`
- verifier salt refresh failing - `Still fake: ...`
- executor command queue delay above a conservative threshold - Push only if tests and dashboard build pass.
- Keep alert scopes executor/service-level unless pair-specific evidence is available. - Let repo workflow deploy; do not manually patch deployments.
- Do not disarm automatically unless an existing safety invariant already does so. - Collect live evidence after deploy:
- deployments on the new image
- dashboard still receives live lifecycle updates
- durable lookup returns at least one recent quote lifecycle row
- an unresolved or missed quote can be inspected by identifier without relying on it being visible in the rolling table
- selected quote UI does not clear on live row movement
### 7. Operator dashboard ## Assumptions
- Surface salt cache state in the system/service view: - Existing durable quote, decision, command, execution result, and outcome rows are sufficient to reconstruct a single quote lifecycle.
- fresh/stale/unavailable - The dashboard can add a small authenticated read endpoint without changing service topology.
- salt age - Client-side debounce plus manual refresh is enough to keep pinned quote investigation fresh without adding a new response-policy or retention setting.
- last refresh time
- last refresh duration
- last refresh error
- Surface executor queue delay alongside existing trade-executor health.
- Preserve existing lifecycle table timing fields and old-row compatibility.
- Avoid presenting salt failures as trade PnL or settled outcomes.
### 8. Tests ## Still Fake
- Add verifier salt cache tests: - Venue-native terminal fill truth remains incomplete where outcome rows only infer not-filled or settlement through existing attribution.
- cached fresh read returns immediately and does not call the loader - Fee-aware realized PnL and broader quote analytics workbench views remain out of scope.
- missing/stale salt returns unavailable without loader call - Historical detail that has already been pruned cannot be reconstructed beyond existing rollups.
- background refresh records success duration and freshness
- background refresh records error without destroying still-fresh salt
- malformed salts fail closed
- Add trade-executor tests or refactorable helper tests:
- slow salt loader cannot block quote command handling
- no cached fresh salt publishes `verifier_salt_unavailable`
- no relay call happens when salt is unavailable
- stale command expiry still happens before salt lookup
- successful command uses cached salt and records `current_salt_source=cache`
- Add runtime health and alert tests for salt and queue warnings.
- Add dashboard/static tests for salt and queue visibility.
### 9. Validation and deploy
- Run targeted tests:
- verifier salt cache
- trade executor
- runtime health / ops sentinel
- dashboard static/operator tests
- Run full `npm test`.
- Build operator dashboard bundle.
- Commit with required workflow metadata.
- Push to `main` so repo workflow deploys.
- Do not manually patch or roll Kubernetes deployments.
### 10. Live evidence after deploy
- Confirm all repo-owned deployments run the new image.
- Query recent trade execution results:
- `current_salt_source`
- `current_salt_ms`
- `command_to_executor_ms`
- `quote_age_at_executor_receipt_ms`
- `result_code`
- `failure_category`
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
- Report command-to-executor p50/p90/p99 after rollout.
- Report salt cache health from trade-executor state or health.
- Confirm recent quote flow still writes decisions, commands, and results.
## Data and persistence
- Use existing executor result payloads for per-command salt and queue timing.
- Add fields additively so old rows remain readable.
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
- Do not add new long-term storage unless required for dashboard or alert correctness.
## Edge cases
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
- Salt cache stale: commands reject quickly, no relay call.
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
- Malformed salt from verifier: fail closed and expose error.
- Existing stale command: reject as stale command before salt lookup.
- Executor paused or disarmed: existing behavior remains.
- Old result rows without salt fields remain visible.
## Concrete implementation order
### Phase 1. Audit and contracts
- Inspect current salt cache and executor usage.
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
- Decide exact result code and payload shape for salt-unavailable rejection.
### Phase 2. Salt cache refactor
- Implement non-blocking cached salt read.
- Implement refresh state fields and in-flight de-duplication.
- Ensure background refresh starts and continues.
- Keep existing cache tests passing with updated contract.
### Phase 3. Executor integration
- Change quote-response hot path to use cached salt only.
- Add salt-unavailable rejection result.
- Add timing/state fields for cache read and unavailable rejection.
- Add regression tests for slow salt loader and no relay call.
### Phase 4. Health, alerts, dashboard
- Expose salt freshness and queue delay in trade-executor state/health.
- Add ops-sentinel/runtime alert tests.
- Add dashboard visibility tests and UI fields.
### Phase 5. Validation and deployment
- Run targeted tests.
- Run full `npm test`.
- Build dashboard bundle.
- Commit and push.
- Collect live post-deploy evidence.
## Test plan
- `test/verifier-salt-cache.test.mjs`
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
- `test/runtime-health.test.mjs`
- `test/ops-sentinel*.test.mjs`
- `test/operator-dashboard*.test.mjs`
- Static config tests proving no new active env knobs are added
- Full `npm test`
- Dashboard build
## Validation checklist against the proof
- Quote-response hot path contains no verifier salt network refresh.
- Slow `current_salt` loader does not cause multi-second command queue delay.
- Missing/stale salt creates durable explicit rejection without relay call.
- Salt freshness and refresh errors are operator-visible.
- Executor queue delay is operator-visible and alertable.
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
- Repo workflow deploys the result.
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.
## Known fakes allowed at start of this turn
- Venue-native terminal fill ids remain unavailable.
- Fee-complete realized PnL remains unavailable.
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.

198
PROOF.md
View file

@ -1,144 +1,80 @@
# Implementation Proof: Verifier salt hot-path removal and executor queue guardrails # Implementation Proof: Quote lifecycle investigator and missed quote follow-up
Status: open Status: open
Opened: 2026-06-12 Opened: 2026-06-12
## Target outcome ## Hypothesis
Quote-response execution must not block on the NEAR verifier `current_salt` RPC. If the operator dashboard turns the quote lifecycle table into a durable quote investigator, an operator can pause or filter the live quote list, select an `Awaiting executor` or otherwise missed quote, and continue following that exact quote through command, executor result, relay failure, outcome, or unresolved state even after the rolling live table moves on.
The executor should use an already-fresh cached signing salt in the hot path, reject quote-response commands quickly and explicitly when no fresh salt is available, and expose enough runtime evidence for the operator to see salt freshness and command queue delay before they turn fresh quotes into stale relay failures. ## Problem Evidence
- The current Quote lifecycle page is a rolling live table of recent rows.
## Why this is the next architecture test - Pausing freezes the displayed table snapshot, including stale `Awaiting executor` evidence.
Live maker timing exposed a real hot-path false path: - When live updates resume, the quote can leave the recent window and become hard to find again.
- The existing expandable row explains the lifecycle only while the row remains in the rendered recent set.
- normal command-to-executor latency is usually milliseconds - This prevents practical investigation of missed quotes such as commands that later become `quote_not_found_or_finished`, `verifier_salt_unavailable`, stale command rejects, or successful submissions.
- at `2026-06-12 14:36 UTC`, one command spent about `7481 ms` refreshing `current_salt`
- the single executor consumer then queued later commands for `7-12s`
- those commands reached the relay after the quote was already gone and failed with `quote_not_found_or_finished`
The trading system cannot treat an external verifier RPC as part of per-quote execution. Execution must outpace observability, but it also must fail closed when signing prerequisites are unavailable.
## Scope ## Scope
- Inspect the current verifier salt cache, trade-executor quote-response hot path, command expiry guard, executor timing payloads, runtime health, ops sentinel alerts, and dashboard service views. - Improve the existing `Quote lifecycle` dashboard view, not a new dashboard product area.
- Add a non-blocking cached-salt API for quote-response execution. - Add a durable quote lookup path by `quote_id`, `decision_id`, or `command_id` using existing history tables.
- Move verifier salt refresh responsibility fully to background refresh and explicit refresh state. - Add a selected or pinned quote investigator that remains visible independently from the rolling table rows.
- Ensure the quote-response hot path never performs `current_salt` RPC or waits on a salt refresh. - Make pause freeze table ordering only; the selected quote investigation must still refresh from durable evidence.
- Reject commands quickly with an explicit durable result when no fresh cached salt is available. - Add search or lookup controls so a quote can be found again after it leaves the recent table.
- Preserve existing command expiry behavior so stale commands fail closed before relay submission. - Add filters for investigation states that matter to missed quotes:
- Surface verifier salt freshness, refresh duration/error, and executor queue delay in runtime health and dashboard/operator surfaces. - awaiting executor
- Add runtime alert coverage for stale or unavailable verifier salt and excessive executor queue delay. - approved but command missing
- Deploy only through the repo workflow and collect live evidence after rollout. - blocked before submit
- submission failed
## Assumptions - quote not found or finished
- A verifier salt can be reused safely within the existing freshness policy enforced by `verifier-salt-cache`. - strategy rejected
- The current salt cache background refresher can be extended rather than replaced. - submitted or awaiting outcome
- A command that cannot be signed because no fresh salt is available should be rejected before relay submission; this is a signing prerequisite failure, not quote skipping to avoid relay errors. - Improve the selected quote detail so `Awaiting executor` clearly shows:
- The executor remains a single quote-response consumer unless implementation evidence shows concurrency is required after salt is removed from the hot path. - command id and command timestamp
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable. - elapsed time since command
- whether an executor result is now stored
## Turn-shaping rules - relay result or block reason if present
- Do not change strategy selection, edge thresholds, notional limits, inventory checks, arming semantics, signer identity, pair enablement, or quote response pricing. - next expected stage when still unresolved
- Do not skip quotes because they may produce relay errors. - Preserve old executor result and lifecycle rows as readable when newer timing or result fields are missing.
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars. - Keep the implementation read-side and dashboard-only unless a small shared lifecycle normalizer change is required.
- Do not hide or relabel relay failures as successful outcomes.
- Do not perform manual rollout, image patching, or cluster repair; deployment must be push-driven through the repo workflow.
- If a salt-related bug is fixed, add regression tests.
- Preserve old executor result rows as readable even if they lack new salt fields.
## Non-goals ## Non-goals
- No new trading strategy. - Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
- No executor concurrency change unless required to prove the salt fix. - Do not skip quotes or alter executor/relay response behavior.
- No change to quote response economics. - Do not add active pair, edge, notional, latency, response-policy, or retention environment variables.
- No new venue or signer. - Do not add a new retention policy or new long-term analytics store in this turn.
- No automatic disarming based on salt health unless an existing safety path already does so. - Do not build the full remaining analytics workbench: quote volume windows, size-bucket distributions, oracle-deviation views, and matched-only PnL analytics remain backlog work.
- No realized PnL or fee attribution. - Do not claim submitted quote responses are completed trades unless durable outcome and settlement evidence support that claim.
- Do not use manual kubectl rollout, image patching, or ad hoc production repair.
## Required operator behavior ## Definition of Done
- The Quote lifecycle page has a stable selected or pinned quote investigator separate from the rolling table rows.
### Salt Truth - Selecting an `Awaiting executor` row lets the operator continue following that quote after the table is paused, resumed, filtered, or updated.
The operator must be able to see: - A selected quote can be refreshed or auto-refreshed from durable evidence even when it is no longer present in `recent_lifecycle_rows`.
- Lookup by `quote_id`, `decision_id`, or `command_id` returns the same normalized lifecycle shape used by the table, including stage timestamps, reason text, timing, command, execution result, and outcome fields when present.
- whether the verifier salt cache is fresh - Problem-state filters make missed quote rows easy to isolate without hiding the selected quote investigation.
- current salt age - Rows with missing old fields render as unavailable instead of crashing or disappearing.
- last refresh start and completion time - Bug regression coverage proves:
- last refresh duration - a selected quote remains inspectable after it leaves the recent live list
- whether a refresh is in flight - an `Awaiting executor` row can later refresh into a terminal executor or relay result
- last refresh error, if any - lookup by quote, decision, and command identifiers uses durable evidence and does not rely only on the current client-side recent rows
- count or timestamp of commands rejected because no fresh salt was available - Targeted dashboard tests pass.
### Queue Truth
The operator must be able to see:
- recent command-to-executor latency
- whether queue delay is warning or critical
- recent executor result timing including salt source and salt lookup duration
- whether quote failures cluster after queue delay rather than relay latency
### Execution Truth
For each quote-response result:
- cached salt use should be explicit in `executor_timing.current_salt_source`
- `current_salt_source=refresh` must not appear from the quote-response hot path after this turn
- no relay call should be attempted when a fresh cached salt is unavailable
- missing or stale salt must produce a durable rejected result with an explicit result code
## Semantic invariants
- Quote-response signing prerequisites are fail-closed.
- No fresh cached salt means no relay submission.
- Hot-path salt lookup is local and bounded; it never performs network I/O.
- Background salt refresh failures are visible but do not block already-fresh cached salt usage.
- Stale commands are still rejected before relay submission.
- Queue delay and relay latency remain separate timing concepts.
- Old executor result rows remain readable.
## Definition of done
- `verifier-salt-cache` exposes a non-blocking fresh-salt read path for hot execution.
- Background salt refresh maintains cache state and exposes freshness, duration, in-flight state, and errors.
- Trade-executor quote-response handling uses only the non-blocking cached-salt path.
- Trade-executor publishes durable rejected results when no fresh cached salt is available.
- Executor timing payloads distinguish cached salt, unavailable salt, and historical refresh rows.
- Runtime health and dashboard expose salt freshness and executor queue delay.
- Ops sentinel or runtime alert tests cover stale/unavailable salt and excessive queue delay.
- Regression tests prove slow verifier salt RPC cannot block quote-response command handling.
- Full `npm test` passes. - Full `npm test` passes.
- Dashboard bundle build passes. - The operator dashboard bundle builds.
- The result is deployed through repo workflow. - The planning and implementation commits include `Proof:`, `Assumptions:`, and `Still fake:`.
- Live post-deploy evidence shows quote flow continues and command-to-executor latency is not dominated by salt refresh. - If tests pass, deploy only through the repo push workflow and collect live evidence:
- repo-owned deployments are on the new image
- Quote lifecycle still receives live rows
- at least one durable quote lookup returns a lifecycle row
- a missed or pending quote can be inspected by identifier after it is not relied on as a visible recent row
## Validation evidence required ## Failure Conditions
- Unit tests for non-blocking cached-salt reads. - Pause still freezes the only available evidence for a selected quote.
- Unit tests proving background refresh records duration/error and does not require hot-path waiting. - A selected quote cannot be found once it leaves the rolling recent table.
- Trade-executor tests proving a slow `current_salt` RPC does not delay command rejection or handling. - The lookup path searches only the browser's current rows instead of durable history.
- Tests proving no relay submission is attempted when cached salt is missing/stale. - The UI improves labels but still cannot answer what happened after `Awaiting executor`.
- Tests proving stale command expiry still wins before signing. - Missed-quote filters hide the selected investigation or make old rows unreadable.
- Runtime health or ops sentinel tests for stale salt and queue-delay warnings. - The implementation changes trading behavior or strategy economics.
- Dashboard/static tests for salt freshness and queue-delay visibility. - No regression test covers the row-follow-up bug.
- Full `npm test`.
- Dashboard bundle build.
- Live evidence after deployment:
- trade-executor deployment on the new image
- recent executor results with `current_salt_source=cache` or explicit salt-unavailable rejection, not hot-path `refresh`
- command-to-executor p50/p90/p99 after rollout
- salt cache freshness state
- recent quote flow still writing decisions, commands, and results
## Failure conditions ## Real Versus Fake
- Any quote-response command performs verifier `current_salt` RPC in the hot path. - Real in scope: dashboard investigation backed by durable quote, decision, command, executor result, and outcome rows already stored by repo-owned services.
- A slow salt refresh queues later commands for seconds. - Still fake or incomplete: venue-native final fill truth, fee-aware realized PnL, broader quote analytics workbench, and historical aggregate drilldowns beyond the existing retained lifecycle detail and rollups.
- Missing or stale salt leads to relay submission instead of fail-closed rejection.
- Dashboard or health hides stale salt or queue delay.
- The implementation changes strategy behavior, pricing, inventory policy, or pair enablement.
- Deployment requires manual cluster reconciliation.
## Current real before this turn
- Trade-executor has a verifier salt cache and records executor timing.
- Live evidence showed normal command-to-executor p50 around `11 ms`, but p99 near `9954 ms` during a salt refresh stall.
- The slow rows showed `current_salt_source=refresh` with `current_salt_ms` around `7481 ms` and `5144 ms`.
- Subsequent queued rows reached the executor after `7-12s` and often failed with `quote_not_found_or_finished`.
- Command expiry and maker timing fields already exist, but salt freshness and queue-delay guardrails are not operator-visible enough.
## Deliberately not built by this proof
- Executor worker pool or broad concurrency model unless the non-blocking salt path is insufficient.
- Fee-complete realized PnL.
- Venue-native terminal fill ids.
- Automatic strategy changes based on salt or queue metrics.

View file

@ -0,0 +1,235 @@
# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails
Status: open
Opened: 2026-06-12
## Goal
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.
## Selected backlog items
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.
## Design rules
- This turn is execution latency and visibility only.
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
- Do not skip quotes because relay errors are likely.
- Do not submit a quote response if signing prerequisites are unavailable.
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
- Keep old executor result rows readable.
- Deploy only through repo workflow.
## Problem statement
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing.
Live evidence showed this can block for seconds:
- `ae824f64-ddfd-4310-a44f-69c401fc8ad7`
- `current_salt_source=refresh`
- `current_salt_ms=7481.458`
- executor received the command quickly, then blocked inside salt refresh
- following commands waited in the single executor consumer queue:
- `command_to_executor_ms` around `7.5s`
- some later rows around `10-12s`
- most failed with `quote_not_found_or_finished`
External verifier latency must not sit on the per-quote execution critical path.
## Backend changes
### 1. Inspect current salt and executor paths
- Inspect:
- `src/core/verifier-salt-cache.mjs`
- `src/apps/trade-executor.mjs`
- executor command expiry helpers
- executor timing payload construction
- runtime health and service snapshot summaries
- ops sentinel alert generation
- operator dashboard service/system views
- existing verifier salt cache and executor tests
- Identify every call path that can refresh verifier salt.
- Confirm which call paths are quote-response hot path versus background or operator-triggered work.
### 2. Define salt cache contract
- Add or refine a cache API with separate responsibilities:
- background refresh may perform network I/O
- hot-path read must never perform network I/O
- Candidate API:
- `start()` starts periodic refresh and immediate refresh attempt
- `refreshNow()` performs explicit refresh with in-flight de-duplication
- `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O
- `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
- Validate malformed salt before it can enter the cache.
### 3. Background refresh behavior
- Ensure startup kicks off refresh without making the executor command handler wait on it.
- Refresh periodically before the salt becomes stale.
- If refresh fails:
- keep a still-fresh cached salt usable until its age limit expires
- expose last error and stale/unavailable state
- do not block command handling
- Record:
- `last_refresh_started_at`
- `last_refresh_completed_at`
- `last_refresh_duration_ms`
- `last_refresh_error`
- `refresh_in_flight`
- `salt_age_ms`
- `salt_fresh`
### 4. Trade-executor hot path
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read.
- If no fresh cached salt is available:
- publish a durable trade execution result
- use status `rejected`
- use explicit result code such as `verifier_salt_unavailable`
- include salt cache state summary in the result payload
- do not call `buildQuoteResponseSubmission`
- do not call the relay
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`.
- Keep idempotency and executor state-store semantics intact.
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.
### 5. Queue delay guardrails
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`.
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
- Add thresholded runtime health fields:
- recent max command-to-executor
- p90 or bucketed queue delay if practical
- latest command queue delay warning reason
- Do not use queue delay to change strategy selection or pricing in this turn.
### 6. Runtime health and alerts
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status.
- Add ops-sentinel warning alerts for:
- verifier salt stale or unavailable
- verifier salt refresh failing
- executor command queue delay above a conservative threshold
- Keep alert scopes executor/service-level unless pair-specific evidence is available.
- Do not disarm automatically unless an existing safety invariant already does so.
### 7. Operator dashboard
- Surface salt cache state in the system/service view:
- fresh/stale/unavailable
- salt age
- last refresh time
- last refresh duration
- last refresh error
- Surface executor queue delay alongside existing trade-executor health.
- Preserve existing lifecycle table timing fields and old-row compatibility.
- Avoid presenting salt failures as trade PnL or settled outcomes.
### 8. Tests
- Add verifier salt cache tests:
- cached fresh read returns immediately and does not call the loader
- missing/stale salt returns unavailable without loader call
- background refresh records success duration and freshness
- background refresh records error without destroying still-fresh salt
- malformed salts fail closed
- Add trade-executor tests or refactorable helper tests:
- slow salt loader cannot block quote command handling
- no cached fresh salt publishes `verifier_salt_unavailable`
- no relay call happens when salt is unavailable
- stale command expiry still happens before salt lookup
- successful command uses cached salt and records `current_salt_source=cache`
- Add runtime health and alert tests for salt and queue warnings.
- Add dashboard/static tests for salt and queue visibility.
### 9. Validation and deploy
- Run targeted tests:
- verifier salt cache
- trade executor
- runtime health / ops sentinel
- dashboard static/operator tests
- Run full `npm test`.
- Build operator dashboard bundle.
- Commit with required workflow metadata.
- Push to `main` so repo workflow deploys.
- Do not manually patch or roll Kubernetes deployments.
### 10. Live evidence after deploy
- Confirm all repo-owned deployments run the new image.
- Query recent trade execution results:
- `current_salt_source`
- `current_salt_ms`
- `command_to_executor_ms`
- `quote_age_at_executor_receipt_ms`
- `result_code`
- `failure_category`
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
- Report command-to-executor p50/p90/p99 after rollout.
- Report salt cache health from trade-executor state or health.
- Confirm recent quote flow still writes decisions, commands, and results.
## Data and persistence
- Use existing executor result payloads for per-command salt and queue timing.
- Add fields additively so old rows remain readable.
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
- Do not add new long-term storage unless required for dashboard or alert correctness.
## Edge cases
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
- Salt cache stale: commands reject quickly, no relay call.
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
- Malformed salt from verifier: fail closed and expose error.
- Existing stale command: reject as stale command before salt lookup.
- Executor paused or disarmed: existing behavior remains.
- Old result rows without salt fields remain visible.
## Concrete implementation order
### Phase 1. Audit and contracts
- Inspect current salt cache and executor usage.
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
- Decide exact result code and payload shape for salt-unavailable rejection.
### Phase 2. Salt cache refactor
- Implement non-blocking cached salt read.
- Implement refresh state fields and in-flight de-duplication.
- Ensure background refresh starts and continues.
- Keep existing cache tests passing with updated contract.
### Phase 3. Executor integration
- Change quote-response hot path to use cached salt only.
- Add salt-unavailable rejection result.
- Add timing/state fields for cache read and unavailable rejection.
- Add regression tests for slow salt loader and no relay call.
### Phase 4. Health, alerts, dashboard
- Expose salt freshness and queue delay in trade-executor state/health.
- Add ops-sentinel/runtime alert tests.
- Add dashboard visibility tests and UI fields.
### Phase 5. Validation and deployment
- Run targeted tests.
- Run full `npm test`.
- Build dashboard bundle.
- Commit and push.
- Collect live post-deploy evidence.
## Test plan
- `test/verifier-salt-cache.test.mjs`
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
- `test/runtime-health.test.mjs`
- `test/ops-sentinel*.test.mjs`
- `test/operator-dashboard*.test.mjs`
- Static config tests proving no new active env knobs are added
- Full `npm test`
- Dashboard build
## Validation checklist against the proof
- Quote-response hot path contains no verifier salt network refresh.
- Slow `current_salt` loader does not cause multi-second command queue delay.
- Missing/stale salt creates durable explicit rejection without relay call.
- Salt freshness and refresh errors are operator-visible.
- Executor queue delay is operator-visible and alertable.
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
- Repo workflow deploys the result.
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.
## Known fakes allowed at start of this turn
- Venue-native terminal fill ids remain unavailable.
- Fee-complete realized PnL remains unavailable.
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.

View file

@ -0,0 +1,144 @@
# Implementation Proof: Verifier salt hot-path removal and executor queue guardrails
Status: open
Opened: 2026-06-12
## Target outcome
Quote-response execution must not block on the NEAR verifier `current_salt` RPC.
The executor should use an already-fresh cached signing salt in the hot path, reject quote-response commands quickly and explicitly when no fresh salt is available, and expose enough runtime evidence for the operator to see salt freshness and command queue delay before they turn fresh quotes into stale relay failures.
## Why this is the next architecture test
Live maker timing exposed a real hot-path false path:
- normal command-to-executor latency is usually milliseconds
- at `2026-06-12 14:36 UTC`, one command spent about `7481 ms` refreshing `current_salt`
- the single executor consumer then queued later commands for `7-12s`
- those commands reached the relay after the quote was already gone and failed with `quote_not_found_or_finished`
The trading system cannot treat an external verifier RPC as part of per-quote execution. Execution must outpace observability, but it also must fail closed when signing prerequisites are unavailable.
## Scope
- Inspect the current verifier salt cache, trade-executor quote-response hot path, command expiry guard, executor timing payloads, runtime health, ops sentinel alerts, and dashboard service views.
- Add a non-blocking cached-salt API for quote-response execution.
- Move verifier salt refresh responsibility fully to background refresh and explicit refresh state.
- Ensure the quote-response hot path never performs `current_salt` RPC or waits on a salt refresh.
- Reject commands quickly with an explicit durable result when no fresh cached salt is available.
- Preserve existing command expiry behavior so stale commands fail closed before relay submission.
- Surface verifier salt freshness, refresh duration/error, and executor queue delay in runtime health and dashboard/operator surfaces.
- Add runtime alert coverage for stale or unavailable verifier salt and excessive executor queue delay.
- Deploy only through the repo workflow and collect live evidence after rollout.
## Assumptions
- A verifier salt can be reused safely within the existing freshness policy enforced by `verifier-salt-cache`.
- The current salt cache background refresher can be extended rather than replaced.
- A command that cannot be signed because no fresh salt is available should be rejected before relay submission; this is a signing prerequisite failure, not quote skipping to avoid relay errors.
- The executor remains a single quote-response consumer unless implementation evidence shows concurrency is required after salt is removed from the hot path.
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
## Turn-shaping rules
- Do not change strategy selection, edge thresholds, notional limits, inventory checks, arming semantics, signer identity, pair enablement, or quote response pricing.
- Do not skip quotes because they may produce relay errors.
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
- Do not hide or relabel relay failures as successful outcomes.
- Do not perform manual rollout, image patching, or cluster repair; deployment must be push-driven through the repo workflow.
- If a salt-related bug is fixed, add regression tests.
- Preserve old executor result rows as readable even if they lack new salt fields.
## Non-goals
- No new trading strategy.
- No executor concurrency change unless required to prove the salt fix.
- No change to quote response economics.
- No new venue or signer.
- No automatic disarming based on salt health unless an existing safety path already does so.
- No realized PnL or fee attribution.
## Required operator behavior
### Salt Truth
The operator must be able to see:
- whether the verifier salt cache is fresh
- current salt age
- last refresh start and completion time
- last refresh duration
- whether a refresh is in flight
- last refresh error, if any
- count or timestamp of commands rejected because no fresh salt was available
### Queue Truth
The operator must be able to see:
- recent command-to-executor latency
- whether queue delay is warning or critical
- recent executor result timing including salt source and salt lookup duration
- whether quote failures cluster after queue delay rather than relay latency
### Execution Truth
For each quote-response result:
- cached salt use should be explicit in `executor_timing.current_salt_source`
- `current_salt_source=refresh` must not appear from the quote-response hot path after this turn
- no relay call should be attempted when a fresh cached salt is unavailable
- missing or stale salt must produce a durable rejected result with an explicit result code
## Semantic invariants
- Quote-response signing prerequisites are fail-closed.
- No fresh cached salt means no relay submission.
- Hot-path salt lookup is local and bounded; it never performs network I/O.
- Background salt refresh failures are visible but do not block already-fresh cached salt usage.
- Stale commands are still rejected before relay submission.
- Queue delay and relay latency remain separate timing concepts.
- Old executor result rows remain readable.
## Definition of done
- `verifier-salt-cache` exposes a non-blocking fresh-salt read path for hot execution.
- Background salt refresh maintains cache state and exposes freshness, duration, in-flight state, and errors.
- Trade-executor quote-response handling uses only the non-blocking cached-salt path.
- Trade-executor publishes durable rejected results when no fresh cached salt is available.
- Executor timing payloads distinguish cached salt, unavailable salt, and historical refresh rows.
- Runtime health and dashboard expose salt freshness and executor queue delay.
- Ops sentinel or runtime alert tests cover stale/unavailable salt and excessive queue delay.
- Regression tests prove slow verifier salt RPC cannot block quote-response command handling.
- Full `npm test` passes.
- Dashboard bundle build passes.
- The result is deployed through repo workflow.
- Live post-deploy evidence shows quote flow continues and command-to-executor latency is not dominated by salt refresh.
## Validation evidence required
- Unit tests for non-blocking cached-salt reads.
- Unit tests proving background refresh records duration/error and does not require hot-path waiting.
- Trade-executor tests proving a slow `current_salt` RPC does not delay command rejection or handling.
- Tests proving no relay submission is attempted when cached salt is missing/stale.
- Tests proving stale command expiry still wins before signing.
- Runtime health or ops sentinel tests for stale salt and queue-delay warnings.
- Dashboard/static tests for salt freshness and queue-delay visibility.
- Full `npm test`.
- Dashboard bundle build.
- Live evidence after deployment:
- trade-executor deployment on the new image
- recent executor results with `current_salt_source=cache` or explicit salt-unavailable rejection, not hot-path `refresh`
- command-to-executor p50/p90/p99 after rollout
- salt cache freshness state
- recent quote flow still writing decisions, commands, and results
## Failure conditions
- Any quote-response command performs verifier `current_salt` RPC in the hot path.
- A slow salt refresh queues later commands for seconds.
- Missing or stale salt leads to relay submission instead of fail-closed rejection.
- Dashboard or health hides stale salt or queue delay.
- The implementation changes strategy behavior, pricing, inventory policy, or pair enablement.
- Deployment requires manual cluster reconciliation.
## Current real before this turn
- Trade-executor has a verifier salt cache and records executor timing.
- Live evidence showed normal command-to-executor p50 around `11 ms`, but p99 near `9954 ms` during a salt refresh stall.
- The slow rows showed `current_salt_source=refresh` with `current_salt_ms` around `7481 ms` and `5144 ms`.
- Subsequent queued rows reached the executor after `7-12s` and often failed with `quote_not_found_or_finished`.
- Command expiry and maker timing fields already exist, but salt freshness and queue-delay guardrails are not operator-visible enough.
## Deliberately not built by this proof
- Executor worker pool or broad concurrency model unless the non-blocking salt path is insufficient.
- Fee-complete realized PnL.
- Venue-native terminal fill ids.
- Automatic strategy changes based on salt or queue metrics.

View file

@ -14,6 +14,7 @@ import {
buildDashboardBootstrap, buildDashboardBootstrap,
buildDashboardControlErrorResponse, buildDashboardControlErrorResponse,
buildLiveQuoteLifecycleRows, buildLiveQuoteLifecycleRows,
buildQuoteLifecycleLookupResponse,
buildLiveStatusBar, buildLiveStatusBar,
createDashboardLiveState, createDashboardLiveState,
isDashboardWebSocketBackpressured, isDashboardWebSocketBackpressured,
@ -44,6 +45,7 @@ import {
loadLatestMarketPrice, loadLatestMarketPrice,
loadLatestPortfolioMetric, loadLatestPortfolioMetric,
loadPairConfigSummary, loadPairConfigSummary,
loadQuoteLifecycleLookupEvidence,
loadQuoteLifecycleRetentionSummary, loadQuoteLifecycleRetentionSummary,
loadRecentAlertTransitions, loadRecentAlertTransitions,
loadRecentDepositStatuses, loadRecentDepositStatuses,
@ -378,6 +380,38 @@ async function handleApiRequest({ req, res, url, auth }) {
return sendJson(res, 200, submissionPage); return sendJson(res, 200, submissionPage);
} }
const lifecycleLookupMatch = req.method === 'GET'
? url.pathname.match(/^\/api\/strategy\/quote-lifecycle\/(.+)$/)
: null;
if (lifecycleLookupMatch) {
const identifier = decodeURIComponent(lifecycleLookupMatch[1] || '').trim();
if (!identifier) {
return sendJson(res, 400, {
ok: false,
error: 'quote_lifecycle_identifier_required',
lifecycle_row: null,
lookup: {
requested_identifier: '',
matched_identifier_type: null,
found: false,
lookup_at: new Date().toISOString(),
latest_stage_at: null,
matched_identifiers: {
quote_ids: [],
decision_ids: [],
command_ids: [],
},
},
});
}
const payload = await loadQuoteLifecycleLookupPayload({ identifier });
return sendJson(res, payload.ok ? 200 : 404, {
...payload,
...(payload.ok ? {} : { error: 'quote_lifecycle_not_found' }),
});
}
const controlMatch = req.method === 'POST' const controlMatch = req.method === 'POST'
? url.pathname.match(/^\/api\/control\/([^/]+)\/([^/]+)$/) ? url.pathname.match(/^\/api\/control\/([^/]+)\/([^/]+)$/)
: null; : null;
@ -597,6 +631,19 @@ async function loadBootstrapPayload({ auth, page, pageSize }) {
return payload; return payload;
} }
async function loadQuoteLifecycleLookupPayload({ identifier }) {
const tradingConfig = await tradingConfigStore.forceRefresh();
const runtimeConfig = buildRuntimeConfig(tradingConfig);
const evidence = await loadQuoteLifecycleLookupEvidence(pool, {
identifier,
});
return buildQuoteLifecycleLookupResponse({
config: runtimeConfig,
identifier,
evidence,
});
}
async function loadServiceSnapshots() { async function loadServiceSnapshots() {
const services = listDashboardServices(config); const services = listDashboardServices(config);
return Promise.all(services.map((service) => loadServiceSnapshot(service))); return Promise.all(services.map((service) => loadServiceSnapshot(service)));

View file

@ -1130,6 +1130,7 @@ const HUMAN_REASON_TEXT = {
pending_deposit_not_credited: 'Funding is not credited yet.', pending_deposit_not_credited: 'Funding is not credited yet.',
pending_outbound_reserved: 'Pending outbound inventory is reserved.', pending_outbound_reserved: 'Pending outbound inventory is reserved.',
quote_expired: 'Quote expired.', quote_expired: 'Quote expired.',
quote_not_found_or_finished: 'Relay reported the quote as not found or already finished.',
quote_response_ack: 'Quote response acknowledged by the relay.', quote_response_ack: 'Quote response acknowledged by the relay.',
quote_response_ok: 'Quote response accepted by the relay.', quote_response_ok: 'Quote response accepted by the relay.',
awaiting_outcome: 'No durable venue outcome is recorded yet.', awaiting_outcome: 'No durable venue outcome is recorded yet.',
@ -1425,7 +1426,11 @@ function finalizeLifecycleRow(row) {
} else if (execution?.status === 'failed') { } else if (execution?.status === 'failed') {
lifecycle_state = 'failed'; lifecycle_state = 'failed';
lifecycle_label = 'Submission failed'; lifecycle_label = 'Submission failed';
reason_code = normalizeLifecycleToken(execution?.result_code || 'submission_failed'); reason_code = normalizeLifecycleToken(
execution?.failure_category
|| execution?.result_code
|| 'submission_failed',
);
reason_text = buildExecutionFailureText(execution, reason_code); reason_text = buildExecutionFailureText(execution, reason_code);
} else if (execution?.status === 'rejected') { } else if (execution?.status === 'rejected') {
lifecycle_state = 'blocked'; lifecycle_state = 'blocked';
@ -1504,6 +1509,100 @@ function finalizeLifecycleRow(row) {
}; };
} }
export function buildQuoteLifecycleLookupResponse({
config,
identifier,
evidence = {},
now = new Date().toISOString(),
} = {}) {
const requestedIdentifier = String(identifier || '').trim();
const rows = deriveQuoteLifecycleRows({
recentQuotes: evidence.recentQuotes || [],
recentTradeDecisions: evidence.recentTradeDecisions || [],
recentExecuteTradeCommands: evidence.recentExecuteTradeCommands || [],
recentExecutionResults: evidence.recentExecutionResults || [],
recentQuoteOutcomes: evidence.recentQuoteOutcomes || [],
limit: null,
}).map((row) => enrichLifecycleRowForUi({ config, row }));
const lifecycleRow = selectLookupLifecycleRow(rows, requestedIdentifier);
const found = Boolean(lifecycleRow);
const lookup = {
requested_identifier: requestedIdentifier,
matched_identifier_type: found
? inferLookupIdentifierType(requestedIdentifier, lifecycleRow, evidence)
: null,
found,
lookup_at: evidence.lookup_at || now,
latest_stage_at: lifecycleRow?.latest_stage_at || null,
matched_identifiers: found
? collectLifecycleIdentifiers(lifecycleRow, evidence.matched_identifiers)
: normalizeMatchedIdentifiers(evidence.matched_identifiers),
};
return {
ok: found,
lifecycle_row: lifecycleRow || null,
lookup,
};
}
function selectLookupLifecycleRow(rows, identifier) {
if (!rows.length) return null;
const normalizedIdentifier = String(identifier || '').trim();
if (!normalizedIdentifier) return rows[0] || null;
return rows.find((row) => lifecycleRowHasIdentifier(row, normalizedIdentifier))
|| rows[0]
|| null;
}
function lifecycleRowHasIdentifier(row, identifier) {
return row?.quote_id === identifier
|| row?.decision_id === identifier
|| row?.command_id === identifier;
}
function inferLookupIdentifierType(identifier, row, evidence = {}) {
if (evidence.matched_identifier_type) return evidence.matched_identifier_type;
if (row?.quote_id === identifier) return 'quote_id';
if (row?.decision_id === identifier) return 'decision_id';
if (row?.command_id === identifier) return 'command_id';
return 'linked_identifier';
}
function collectLifecycleIdentifiers(row, matchedIdentifiers = null) {
const normalized = normalizeMatchedIdentifiers(matchedIdentifiers);
return {
quote_ids: uniqueStrings([
...normalized.quote_ids,
row?.quote_id,
]),
decision_ids: uniqueStrings([
...normalized.decision_ids,
row?.decision_id,
]),
command_ids: uniqueStrings([
...normalized.command_ids,
row?.command_id,
]),
};
}
function normalizeMatchedIdentifiers(value = null) {
return {
quote_ids: uniqueStrings(value?.quote_ids),
decision_ids: uniqueStrings(value?.decision_ids),
command_ids: uniqueStrings(value?.command_ids),
};
}
function uniqueStrings(values = []) {
return [...new Set(
(values || [])
.map((value) => String(value || '').trim())
.filter(Boolean),
)];
}
function buildPolicySkipText(decision, reasonCode) { function buildPolicySkipText(decision, reasonCode) {
const base = humanizeReasonCode(reasonCode, 'Policy skipped the response.'); const base = humanizeReasonCode(reasonCode, 'Policy skipped the response.');
const policy = decision?.response_policy || {}; const policy = decision?.response_policy || {};

View file

@ -3868,6 +3868,354 @@ export async function loadRecentQuoteOutcomes(pool, { limit = 200 } = {}) {
return result.rows.map(normalizeQuoteOutcomeRow); return result.rows.map(normalizeQuoteOutcomeRow);
} }
export async function loadQuoteLifecycleLookupEvidence(pool, { identifier } = {}) {
const requestedIdentifier = String(identifier || '').trim();
const lookupAt = new Date().toISOString();
if (!requestedIdentifier) {
return buildQuoteLifecycleLookupEvidence({
requestedIdentifier,
lookupAt,
});
}
const candidatesResult = await pool.query(
`
WITH candidates AS (
SELECT
'quote_id' AS matched_identifier_type,
COALESCE(quote_id, payload->>'quote_id') AS quote_id,
payload->>'decision_id' AS decision_id,
payload->>'command_id' AS command_id,
COALESCE(observed_at, ingested_at) AS matched_at
FROM swap_demand_events
WHERE quote_id = $1
OR payload->>'quote_id' = $1
UNION ALL
SELECT
CASE
WHEN quote_id = $1 OR payload->>'quote_id' = $1 THEN 'quote_id'
WHEN decision_key = $1 OR payload->>'decision_id' = $1 THEN 'decision_id'
WHEN payload->>'command_id' = $1 THEN 'command_id'
ELSE 'linked_identifier'
END AS matched_identifier_type,
COALESCE(quote_id, payload->>'quote_id') AS quote_id,
COALESCE(payload->>'decision_id', decision_key) AS decision_id,
payload->>'command_id' AS command_id,
COALESCE(observed_at, ingested_at) AS matched_at
FROM trade_decisions
WHERE quote_id = $1
OR decision_key = $1
OR payload->>'quote_id' = $1
OR payload->>'decision_id' = $1
OR payload->>'command_id' = $1
UNION ALL
SELECT
CASE
WHEN quote_id = $1 OR payload->>'quote_id' = $1 THEN 'quote_id'
WHEN payload->>'decision_id' = $1 THEN 'decision_id'
WHEN decision_key = $1 OR payload->>'command_id' = $1 THEN 'command_id'
ELSE 'linked_identifier'
END AS matched_identifier_type,
COALESCE(quote_id, payload->>'quote_id') AS quote_id,
payload->>'decision_id' AS decision_id,
COALESCE(payload->>'command_id', decision_key) AS command_id,
COALESCE(observed_at, ingested_at) AS matched_at
FROM execute_trade_commands
WHERE quote_id = $1
OR decision_key = $1
OR payload->>'quote_id' = $1
OR payload->>'decision_id' = $1
OR payload->>'command_id' = $1
UNION ALL
SELECT
CASE
WHEN quote_id = $1 OR payload->>'quote_id' = $1 THEN 'quote_id'
WHEN payload->>'decision_id' = $1 THEN 'decision_id'
WHEN decision_key = $1 OR payload->>'command_id' = $1 THEN 'command_id'
ELSE 'linked_identifier'
END AS matched_identifier_type,
COALESCE(quote_id, payload->>'quote_id') AS quote_id,
payload->>'decision_id' AS decision_id,
COALESCE(payload->>'command_id', decision_key) AS command_id,
COALESCE(observed_at, ingested_at) AS matched_at
FROM trade_execution_results
WHERE quote_id = $1
OR decision_key = $1
OR payload->>'quote_id' = $1
OR payload->>'decision_id' = $1
OR payload->>'command_id' = $1
UNION ALL
SELECT
CASE
WHEN quote_id = $1 OR payload->>'quote_id' = $1 THEN 'quote_id'
WHEN decision_id = $1 OR payload->>'decision_id' = $1 THEN 'decision_id'
WHEN command_id = $1 OR payload->>'command_id' = $1 THEN 'command_id'
ELSE 'linked_identifier'
END AS matched_identifier_type,
COALESCE(quote_id, payload->>'quote_id') AS quote_id,
COALESCE(decision_id, payload->>'decision_id') AS decision_id,
COALESCE(command_id, payload->>'command_id') AS command_id,
COALESCE(outcome_observed_at, submitted_at, command_at, computed_at) AS matched_at
FROM ${QUOTE_OUTCOMES_TABLE}
WHERE quote_id = $1
OR decision_id = $1
OR command_id = $1
OR payload->>'quote_id' = $1
OR payload->>'decision_id' = $1
OR payload->>'command_id' = $1
)
SELECT *
FROM candidates
ORDER BY matched_at DESC NULLS LAST
LIMIT 100
`,
[requestedIdentifier],
);
const identifiers = collectQuoteLifecycleLookupIdentifiers(candidatesResult.rows);
if (!hasLookupIdentifiers(identifiers)) {
return buildQuoteLifecycleLookupEvidence({
requestedIdentifier,
lookupAt,
matchedIdentifierType: null,
matchedIdentifiers: identifiers,
});
}
const [
recentQuotes,
recentTradeDecisions,
recentExecuteTradeCommands,
recentExecutionResults,
recentQuoteOutcomes,
] = await Promise.all([
loadLifecycleLookupQuotes(pool, identifiers),
loadLifecycleLookupTradeDecisions(pool, identifiers),
loadLifecycleLookupExecuteTradeCommands(pool, identifiers),
loadLifecycleLookupExecutionResults(pool, identifiers),
loadLifecycleLookupQuoteOutcomes(pool, identifiers),
]);
return buildQuoteLifecycleLookupEvidence({
requestedIdentifier,
lookupAt,
matchedIdentifierType: inferLookupMatchedIdentifierType(
requestedIdentifier,
candidatesResult.rows,
),
matchedIdentifiers: identifiers,
recentQuotes,
recentTradeDecisions,
recentExecuteTradeCommands,
recentExecutionResults,
recentQuoteOutcomes,
});
}
function buildQuoteLifecycleLookupEvidence({
requestedIdentifier,
lookupAt,
matchedIdentifierType = null,
matchedIdentifiers = null,
recentQuotes = [],
recentTradeDecisions = [],
recentExecuteTradeCommands = [],
recentExecutionResults = [],
recentQuoteOutcomes = [],
} = {}) {
return {
found: Boolean(
recentQuotes.length
|| recentTradeDecisions.length
|| recentExecuteTradeCommands.length
|| recentExecutionResults.length
|| recentQuoteOutcomes.length,
),
requested_identifier: requestedIdentifier,
matched_identifier_type: matchedIdentifierType,
matched_identifiers: normalizeQuoteLifecycleLookupIdentifiers(matchedIdentifiers),
lookup_at: lookupAt,
recentQuotes,
recentTradeDecisions,
recentExecuteTradeCommands,
recentExecutionResults,
recentQuoteOutcomes,
};
}
function collectQuoteLifecycleLookupIdentifiers(rows = []) {
return normalizeQuoteLifecycleLookupIdentifiers({
quote_ids: rows.map((row) => row.quote_id),
decision_ids: rows.map((row) => row.decision_id),
command_ids: rows.map((row) => row.command_id),
});
}
function normalizeQuoteLifecycleLookupIdentifiers(value = null) {
return {
quote_ids: uniqueTextValues(value?.quote_ids),
decision_ids: uniqueTextValues(value?.decision_ids),
command_ids: uniqueTextValues(value?.command_ids),
};
}
function uniqueTextValues(values = []) {
return [...new Set(
(values || [])
.map((value) => String(value || '').trim())
.filter(Boolean),
)];
}
function hasLookupIdentifiers(identifiers) {
return Boolean(
identifiers?.quote_ids?.length
|| identifiers?.decision_ids?.length
|| identifiers?.command_ids?.length,
);
}
function inferLookupMatchedIdentifierType(identifier, rows = []) {
const requestedIdentifier = String(identifier || '').trim();
const direct = rows.find((row) => (
row.matched_identifier_type
&& (
row.quote_id === requestedIdentifier
|| row.decision_id === requestedIdentifier
|| row.command_id === requestedIdentifier
)
));
return direct?.matched_identifier_type || rows[0]?.matched_identifier_type || null;
}
async function loadLifecycleLookupQuotes(pool, identifiers) {
if (!identifiers.quote_ids.length) return [];
const result = await pool.query(
`
SELECT observed_at, ingested_at, payload
FROM swap_demand_events
WHERE quote_id = ANY($1::text[])
OR payload->>'quote_id' = ANY($1::text[])
ORDER BY COALESCE(observed_at, ingested_at) DESC
LIMIT 20
`,
[identifiers.quote_ids],
);
return result.rows.map(normalizeRecentQuoteRow);
}
async function loadLifecycleLookupTradeDecisions(pool, identifiers) {
const result = await pool.query(
`
SELECT observed_at, ingested_at, payload
FROM trade_decisions
WHERE quote_id = ANY($1::text[])
OR payload->>'quote_id' = ANY($1::text[])
OR decision_key = ANY($2::text[])
OR payload->>'decision_id' = ANY($2::text[])
ORDER BY COALESCE(observed_at, ingested_at) DESC
LIMIT 20
`,
[identifiers.quote_ids, identifiers.decision_ids],
);
return result.rows.map((row) => ({
observed_at: toIsoTimestamp(row.observed_at),
ingested_at: toIsoTimestamp(row.ingested_at),
payload: row.payload,
}));
}
async function loadLifecycleLookupExecuteTradeCommands(pool, identifiers) {
const result = await pool.query(
`
SELECT observed_at, ingested_at, payload
FROM execute_trade_commands
WHERE quote_id = ANY($1::text[])
OR payload->>'quote_id' = ANY($1::text[])
OR payload->>'decision_id' = ANY($2::text[])
OR decision_key = ANY($3::text[])
OR payload->>'command_id' = ANY($3::text[])
ORDER BY COALESCE(observed_at, ingested_at) DESC
LIMIT 20
`,
[identifiers.quote_ids, identifiers.decision_ids, identifiers.command_ids],
);
return result.rows.map((row) => normalizeExecuteTradeCommandRow(row));
}
async function loadLifecycleLookupExecutionResults(pool, identifiers) {
const result = await pool.query(
`
SELECT
r.observed_at AS result_observed_at,
r.ingested_at AS result_ingested_at,
r.payload AS result_payload,
c.ingested_at AS command_ingested_at,
c.payload AS command_payload,
d.payload AS decision_payload,
o.payload AS outcome_payload
FROM trade_execution_results r
LEFT JOIN execute_trade_commands c
ON c.decision_key = r.decision_key
LEFT JOIN trade_decisions d
ON d.decision_key = COALESCE(c.payload->>'decision_id', r.payload->>'decision_id')
LEFT JOIN ${QUOTE_OUTCOMES_TABLE} o
ON o.quote_id = r.quote_id
WHERE r.quote_id = ANY($1::text[])
OR r.payload->>'quote_id' = ANY($1::text[])
OR r.payload->>'decision_id' = ANY($2::text[])
OR r.decision_key = ANY($3::text[])
OR r.payload->>'command_id' = ANY($3::text[])
ORDER BY COALESCE(r.observed_at, r.ingested_at) DESC
LIMIT 20
`,
[identifiers.quote_ids, identifiers.decision_ids, identifiers.command_ids],
);
return result.rows.map((row) => normalizeExecutionResultRow(row));
}
async function loadLifecycleLookupQuoteOutcomes(pool, identifiers) {
const result = await pool.query(
`
SELECT
quote_id,
decision_id,
command_id,
execution_result_status,
execution_result_code,
submitted_at,
command_at,
outcome_status,
outcome_observed_at,
outcome_source,
attribution_status,
attribution_method,
attributed_inventory_delta,
computed_at,
payload
FROM ${QUOTE_OUTCOMES_TABLE}
WHERE quote_id = ANY($1::text[])
OR payload->>'quote_id' = ANY($1::text[])
OR decision_id = ANY($2::text[])
OR payload->>'decision_id' = ANY($2::text[])
OR command_id = ANY($3::text[])
OR payload->>'command_id' = ANY($3::text[])
ORDER BY COALESCE(outcome_observed_at, submitted_at, command_at, computed_at) DESC
LIMIT 20
`,
[identifiers.quote_ids, identifiers.decision_ids, identifiers.command_ids],
);
return result.rows.map(normalizeQuoteOutcomeRow);
}
export async function loadIntentRequestPreflightByIdOrKey(pool, { export async function loadIntentRequestPreflightByIdOrKey(pool, {
requestId = null, requestId = null,
idempotencyKey = null, idempotencyKey = null,

View file

@ -0,0 +1,91 @@
export function lifecycleRowIdentifier(row = null) {
return row?.quote_id || row?.decision_id || row?.command_id || null;
}
export function findLifecycleRowByIdentifier(rows = [], identifier = '') {
const normalizedIdentifier = String(identifier || '').trim();
if (!normalizedIdentifier) return null;
return (rows || []).find((row) => (
row?.quote_id === normalizedIdentifier
|| row?.decision_id === normalizedIdentifier
|| row?.command_id === normalizedIdentifier
)) || null;
}
export function createSelectedLifecycleInvestigation(row, {
source = 'live_row',
lookup = null,
now = Date.now(),
} = {}) {
const identifier = lifecycleRowIdentifier(row);
if (!identifier) return null;
const refreshedAt = toIsoTimestamp(now);
return {
identifier,
row,
source,
lookup,
selected_at: refreshedAt,
refreshed_at: refreshedAt,
error: null,
};
}
export function updateSelectedLifecycleFromLiveRows(selected, rows = [], {
now = Date.now(),
} = {}) {
if (!selected?.identifier) return selected || null;
const liveRow = findLifecycleRowByIdentifier(rows, selected.identifier);
if (!liveRow) return selected;
return {
...selected,
row: liveRow,
source: 'live_row',
refreshed_at: toIsoTimestamp(now),
error: null,
};
}
export function shouldRefreshSelectedFromDurable(selected, rows = []) {
if (!selected?.identifier) return false;
return !findLifecycleRowByIdentifier(rows, selected.identifier);
}
export function updateSelectedLifecycleFromLookup(selected, payload, {
identifier = null,
now = Date.now(),
} = {}) {
const requestedIdentifier = identifier || selected?.identifier || payload?.lookup?.requested_identifier || null;
const refreshedAt = toIsoTimestamp(now);
if (payload?.ok && payload.lifecycle_row) {
return {
...(selected || {}),
identifier: lifecycleRowIdentifier(payload.lifecycle_row) || requestedIdentifier,
row: payload.lifecycle_row,
source: 'durable_lookup',
lookup: payload.lookup || null,
selected_at: selected?.selected_at || refreshedAt,
refreshed_at: refreshedAt,
error: null,
};
}
return {
...(selected || {}),
identifier: requestedIdentifier,
row: selected?.row || null,
source: payload?.error === 'quote_lifecycle_not_found' || payload?.lookup?.found === false
? 'not_found'
: 'lookup_error',
lookup: payload?.lookup || null,
selected_at: selected?.selected_at || refreshedAt,
refreshed_at: refreshedAt,
error: payload?.error || null,
};
}
function toIsoTimestamp(value) {
const timestamp = typeof value === 'number' ? value : Date.parse(value);
if (!Number.isFinite(timestamp)) return new Date().toISOString();
return new Date(timestamp).toISOString();
}

View file

@ -5,6 +5,13 @@ import MetricCard from '../components/MetricCard.jsx';
import Pill from '../components/Pill.jsx'; import Pill from '../components/Pill.jsx';
import TableFrame from '../components/TableFrame.jsx'; import TableFrame from '../components/TableFrame.jsx';
import { formatAgeFromTimestamp, formatBoolean, formatEur, formatTimestamp, truncateMiddle } from '../lib/format.js'; import { formatAgeFromTimestamp, formatBoolean, formatEur, formatTimestamp, truncateMiddle } from '../lib/format.js';
import {
createSelectedLifecycleInvestigation,
findLifecycleRowByIdentifier,
shouldRefreshSelectedFromDurable,
updateSelectedLifecycleFromLiveRows,
updateSelectedLifecycleFromLookup,
} from '../lib/quoteLifecycleInvestigator.js';
const RESPONDED_STATES = new Set(['submitted', 'awaiting_outcome', 'not_filled', 'completed']); const RESPONDED_STATES = new Set(['submitted', 'awaiting_outcome', 'not_filled', 'completed']);
const TRADING_PAIR_MODES = new Set(['maker', 'taker', 'both']); const TRADING_PAIR_MODES = new Set(['maker', 'taker', 'both']);
@ -13,6 +20,17 @@ const COMPETITIVENESS_DETAIL_ROW_COUNT = 8;
const COMPETITIVENESS_LATENCY_ROW_COUNT = 6; const COMPETITIVENESS_LATENCY_ROW_COUNT = 6;
const COMPETITIVENESS_ROLLUP_ROW_COUNT = 8; const COMPETITIVENESS_ROLLUP_ROW_COUNT = 8;
const QUOTE_LIFECYCLE_ROW_COUNT = 20; const QUOTE_LIFECYCLE_ROW_COUNT = 20;
const QUOTE_LIFECYCLE_LOOKUP_DEBOUNCE_MS = 750;
const QUOTE_LIFECYCLE_PROBLEM_FILTERS = [
['all', 'All states'],
['awaiting_executor', 'Awaiting executor'],
['approved_no_command', 'Approved, no command'],
['blocked_before_submit', 'Blocked before submit'],
['submission_failed', 'Submission failed'],
['quote_not_found_or_finished', 'Quote not found or finished'],
['strategy_rejected', 'Strategy rejected'],
['submitted_awaiting_outcome', 'Submitted or awaiting outcome'],
];
async function copyIdentifier(value) { async function copyIdentifier(value) {
if (!value || !navigator?.clipboard?.writeText) return; if (!value || !navigator?.clipboard?.writeText) return;
@ -23,6 +41,22 @@ async function copyIdentifier(value) {
} }
} }
async function fetchQuoteLifecycleLookup(identifier) {
const response = await fetch(`/api/strategy/quote-lifecycle/${encodeURIComponent(identifier)}`, {
headers: {
accept: 'application/json',
},
});
const text = await response.text();
const payload = text.trim() ? JSON.parse(text) : null;
if (response.status === 404 && payload?.error === 'quote_lifecycle_not_found') return payload;
if (!response.ok) {
throw new Error(payload?.error || text.trim() || `HTTP ${response.status}`);
}
return payload;
}
function useNow(intervalMs = 1000) { function useNow(intervalMs = 1000) {
const [now, setNow] = useState(() => Date.now()); const [now, setNow] = useState(() => Date.now());
@ -179,6 +213,86 @@ function isStrategyRejected(item) {
|| String(item?.lifecycle_label || '').toLowerCase() === 'rejected by strategy'; || String(item?.lifecycle_label || '').toLowerCase() === 'rejected by strategy';
} }
function isQuoteNotFoundOrFinished(item) {
return [
item?.reason_code,
item?.execution?.result_code,
item?.execution?.failure_category,
item?.execution?.outcome_reason,
item?.outcome?.outcome_reason,
item?.outcome_reason,
].some((value) => String(value || '').toLowerCase() === 'quote_not_found_or_finished');
}
function matchesLifecycleProblemFilter(item, filter) {
switch (filter) {
case 'awaiting_executor':
return item?.lifecycle_state === 'command_emitted';
case 'approved_no_command':
return item?.lifecycle_state === 'evaluated';
case 'blocked_before_submit':
return item?.lifecycle_state === 'blocked';
case 'submission_failed':
return item?.lifecycle_state === 'failed';
case 'quote_not_found_or_finished':
return isQuoteNotFoundOrFinished(item);
case 'strategy_rejected':
return isStrategyRejected(item);
case 'submitted_awaiting_outcome':
return item?.lifecycle_state === 'submitted' || item?.lifecycle_state === 'awaiting_outcome';
case 'all':
default:
return true;
}
}
function sourceLabel(source) {
if (source === 'live_row') return 'Live row';
if (source === 'durable_lookup') return 'Durable lookup';
if (source === 'not_found') return 'Not found';
if (source === 'lookup_error') return 'Lookup error';
if (source === 'loading') return 'Lookup loading';
return 'No selection';
}
function sourceTone(source) {
if (source === 'not_found') return 'warning';
if (source === 'lookup_error') return 'critical';
if (source === 'live_row' || source === 'durable_lookup') return 'healthy';
return 'unknown';
}
function formatCommandAge(item, now) {
const age = formatAgeFromTimestamp(item?.command_at, now);
return age === 'Unavailable' ? 'Unavailable' : `${age} old`;
}
function executorResultState(item) {
if (item?.execution?.status) return `Recorded: ${plainCodeLabel(item.execution.status)}`;
if (item?.execution_result_at) return 'Recorded';
if (item?.lifecycle_state === 'command_emitted') return 'Not recorded yet';
return 'Unavailable';
}
function nextExpectedStage(item) {
if (item?.lifecycle_state === 'command_emitted') {
return 'Executor result, then relay or outcome evidence';
}
if (item?.lifecycle_state === 'submitted' || item?.lifecycle_state === 'awaiting_outcome') {
return 'Durable outcome and settlement attribution';
}
if (['failed', 'blocked', 'not_filled', 'completed', 'rejected'].includes(item?.lifecycle_state)) {
return 'No later stage is recorded as expected from current evidence';
}
return 'Strategy decision or execute command';
}
function awaitingExecutorBrief(item, now) {
if (item?.lifecycle_state !== 'command_emitted') return null;
const command = item.command_id ? truncateMiddle(item.command_id, 24) : 'Unavailable';
return `Command ${command}; age ${formatCommandAge(item, now)}; executor result ${executorResultState(item)}; next ${nextExpectedStage(item)}`;
}
function StageCard({ title, at, status, children }) { function StageCard({ title, at, status, children }) {
return ( return (
<div className="lifecycle-stage-card"> <div className="lifecycle-stage-card">
@ -239,7 +353,7 @@ function TimingWaterfall({ timing }) {
); );
} }
function LifecycleDetails({ item }) { function LifecycleDetails({ item, now = Date.now() }) {
const executionTiming = formatExecutionTiming(item.execution?.timing); const executionTiming = formatExecutionTiming(item.execution?.timing);
const makerTerms = formatMakerTerms(item.maker_terms); const makerTerms = formatMakerTerms(item.maker_terms);
const inventoryCheck = formatInventoryCheck(item.inventory_check); const inventoryCheck = formatInventoryCheck(item.inventory_check);
@ -262,12 +376,17 @@ function LifecycleDetails({ item }) {
<StageCard at={item.command_at} status={item.command_id ? 'Command recorded' : 'No command'} title="3. Executor command"> <StageCard at={item.command_at} status={item.command_id ? 'Command recorded' : 'No command'} title="3. Executor command">
<IdentifierRow label="Command" value={item.command_id} /> <IdentifierRow label="Command" value={item.command_id} />
<div className="status-subtle">{`Command timestamp: ${formatTimestamp(item.command_at)}`}</div>
<div className="status-subtle">{`Command age: ${formatCommandAge(item, now)}`}</div>
<div className="status-subtle">{`Executor result: ${executorResultState(item)}`}</div>
<div className="status-subtle">{`Next expected: ${nextExpectedStage(item)}`}</div>
<div>{makerTerms || formatTerms(item.submitted_terms)}</div> <div>{makerTerms || formatTerms(item.submitted_terms)}</div>
{makerTerms ? <div className="status-subtle">{formatTerms(item.submitted_terms)}</div> : null} {makerTerms ? <div className="status-subtle">{formatTerms(item.submitted_terms)}</div> : null}
</StageCard> </StageCard>
<StageCard at={item.execution_result_at} status={item.execution?.status || 'No relay result'} title="4. Relay response"> <StageCard at={item.execution_result_at} status={item.execution?.status || 'No relay result'} title="4. Relay response">
<div>{item.execution?.result_code || 'No executor result code stored'}</div> <div>{item.execution?.result_code || 'No executor result code stored'}</div>
{item.execution?.failure_category ? <div className="status-subtle">{plainCodeLabel(item.execution.failure_category)}</div> : null}
{item.execution?.error_message ? <div className="status-subtle">{item.execution.error_message}</div> : null} {item.execution?.error_message ? <div className="status-subtle">{item.execution.error_message}</div> : null}
{!item.execution?.error_message && item.execution?.note ? <div className="status-subtle">{item.execution.note}</div> : null} {!item.execution?.error_message && item.execution?.note ? <div className="status-subtle">{item.execution.note}</div> : null}
{executionTiming ? <div className="status-subtle">{executionTiming}</div> : null} {executionTiming ? <div className="status-subtle">{executionTiming}</div> : null}
@ -294,6 +413,81 @@ function LifecycleDetails({ item }) {
); );
} }
function QuoteLifecycleInvestigator({
selected,
lookupState,
lookupError,
now,
onRefresh,
}) {
const row = selected?.row || null;
const source = lookupState === 'loading' ? 'loading' : selected?.source;
return (
<div className="quote-investigator-panel">
<div className="quote-investigator-head">
<div>
<div className="eyebrow">Pinned quote investigator</div>
<h4>{row?.quote_id ? truncateMiddle(row.quote_id, 46) : selected?.identifier ? truncateMiddle(selected.identifier, 46) : 'No quote selected'}</h4>
<div className="panel-subtitle">
{row
? `${row.lifecycle_label || 'Unavailable'} - ${row.reason_text || 'Unavailable'}`
: selected?.source === 'not_found'
? 'No durable lifecycle evidence matched that identifier.'
: 'Select a row or look up an identifier to inspect durable lifecycle evidence.'}
</div>
</div>
<div className="quote-investigator-actions">
<Pill label={sourceLabel(source)} stateLabel={sourceTone(source)} />
{selected?.refreshed_at ? <Pill label={`Refreshed ${formatRelativeAge(selected.refreshed_at, now)}`} stateLabel="info" /> : null}
<button
className="button secondary"
disabled={!selected?.identifier || lookupState === 'loading'}
onClick={() => onRefresh?.(selected.identifier)}
type="button"
>
Refresh lookup
</button>
</div>
</div>
{lookupError ? <div className="banner error">{lookupError}</div> : null}
{row ? (
<>
<div className="quote-investigator-facts">
<div>
<div className="status-label">Lifecycle</div>
<div className="status-value">{row.lifecycle_label || 'Unavailable'}</div>
<div className="status-subtle mono">{row.reason_code || 'Unavailable'}</div>
</div>
<div>
<div className="status-label">Command age</div>
<div className="status-value">{formatCommandAge(row, now)}</div>
<div className="status-subtle mono">{row.command_id || 'Unavailable'}</div>
</div>
<div>
<div className="status-label">Executor result</div>
<div className="status-value">{executorResultState(row)}</div>
<div className="status-subtle">{formatTimestamp(row.execution_result_at)}</div>
</div>
<div>
<div className="status-label">Next expected</div>
<div className="status-value quote-investigator-next">{nextExpectedStage(row)}</div>
<div className="status-subtle">{formatTimestamp(row.latest_stage_at)}</div>
</div>
</div>
<LifecycleDetails item={row} now={now} />
</>
) : (
<EmptyState>
Durable lifecycle evidence is unavailable for the current identifier.
</EmptyState>
)}
</div>
);
}
function pairDisplayLabel(pairId, pairConfig) { function pairDisplayLabel(pairId, pairConfig) {
const pair = (pairConfig?.pairs || []).find((entry) => ( const pair = (pairConfig?.pairs || []).find((entry) => (
(entry.pair_id || entry.pairId || entry.pair) === pairId (entry.pair_id || entry.pairId || entry.pair) === pairId
@ -661,15 +855,45 @@ function MakerCompetitivenessSection({ summary, pairConfig }) {
function QuoteLifecycleTable({ items }) { function QuoteLifecycleTable({ items }) {
const [expanded, setExpanded] = useState(() => new Set()); const [expanded, setExpanded] = useState(() => new Set());
const [showStrategyRejected, setShowStrategyRejected] = useState(true); const [showStrategyRejected, setShowStrategyRejected] = useState(true);
const [problemFilter, setProblemFilter] = useState('all');
const latestItemsRef = useRef(items || []); const latestItemsRef = useRef(items || []);
const [quoteDisplayPaused, setQuoteDisplayPaused] = useState(false); const [quoteDisplayPaused, setQuoteDisplayPaused] = useState(false);
const [displayItems, setDisplayItems] = useState(() => items || []); const [displayItems, setDisplayItems] = useState(() => items || []);
const [lookupInput, setLookupInput] = useState('');
const [lookupState, setLookupState] = useState('idle');
const [lookupError, setLookupError] = useState(null);
const [selectedInvestigation, setSelectedInvestigation] = useState(null);
const selectedInvestigationRef = useRef(null);
const lookupTimerRef = useRef(null);
const liveNow = useNow(); const liveNow = useNow();
const [displayNow, setDisplayNow] = useState(() => Date.now()); const [displayNow, setDisplayNow] = useState(() => Date.now());
useEffect(() => {
selectedInvestigationRef.current = selectedInvestigation;
}, [selectedInvestigation]);
useEffect(() => () => {
if (lookupTimerRef.current) window.clearTimeout(lookupTimerRef.current);
}, []);
useEffect(() => { useEffect(() => {
latestItemsRef.current = items || []; latestItemsRef.current = items || [];
if (!quoteDisplayPaused) setDisplayItems(items || []); if (!quoteDisplayPaused) setDisplayItems(items || []);
const selected = selectedInvestigationRef.current;
if (!selected?.identifier) return;
const updated = updateSelectedLifecycleFromLiveRows(selected, items || [], {
now: Date.now(),
});
if (updated !== selected) {
setSelectedInvestigation(updated);
setLookupState('live');
setLookupError(null);
return;
}
if (shouldRefreshSelectedFromDurable(selected, items || [])) {
scheduleDurableLookup(selected.identifier);
}
}, [items, quoteDisplayPaused]); }, [items, quoteDisplayPaused]);
useEffect(() => { useEffect(() => {
@ -680,9 +904,17 @@ function QuoteLifecycleTable({ items }) {
() => displayItems.filter((item) => isStrategyRejected(item)).length, () => displayItems.filter((item) => isStrategyRejected(item)).length,
[displayItems], [displayItems],
); );
const problemFilteredItems = useMemo(
() => displayItems.filter((item) => matchesLifecycleProblemFilter(item, problemFilter)),
[displayItems, problemFilter],
);
const visibleItems = useMemo( const visibleItems = useMemo(
() => (showStrategyRejected ? displayItems : displayItems.filter((item) => !isStrategyRejected(item))), () => (
[displayItems, showStrategyRejected], showStrategyRejected || problemFilter === 'strategy_rejected'
? problemFilteredItems
: problemFilteredItems.filter((item) => !isStrategyRejected(item))
),
[problemFilteredItems, problemFilter, showStrategyRejected],
); );
const visibleRows = fixedRows(visibleItems, QUOTE_LIFECYCLE_ROW_COUNT); const visibleRows = fixedRows(visibleItems, QUOTE_LIFECYCLE_ROW_COUNT);
const emptyRowsMessage = !displayItems.length const emptyRowsMessage = !displayItems.length
@ -700,6 +932,77 @@ function QuoteLifecycleTable({ items }) {
}); });
} }
async function performLookup(identifier, { silent = false } = {}) {
const trimmed = String(identifier || '').trim();
if (!trimmed) return;
if (lookupTimerRef.current) {
window.clearTimeout(lookupTimerRef.current);
lookupTimerRef.current = null;
}
if (!silent) setLookupState('loading');
setLookupError(null);
try {
const payload = await fetchQuoteLifecycleLookup(trimmed);
setSelectedInvestigation((current) => updateSelectedLifecycleFromLookup(
current?.identifier === trimmed ? current : { identifier: trimmed, row: null },
payload,
{ identifier: trimmed, now: Date.now() },
));
setLookupState(payload?.ok ? 'found' : 'not_found');
setLookupError(null);
} catch (error) {
const message = error?.message || 'Lookup failed';
setSelectedInvestigation((current) => updateSelectedLifecycleFromLookup(
current?.identifier === trimmed ? current : { identifier: trimmed, row: null },
{
error: message,
lookup: {
requested_identifier: trimmed,
matched_identifier_type: null,
found: false,
lookup_at: new Date().toISOString(),
latest_stage_at: null,
matched_identifiers: {
quote_ids: [],
decision_ids: [],
command_ids: [],
},
},
},
{ identifier: trimmed, now: Date.now() },
));
setLookupState('error');
setLookupError(message);
}
}
function scheduleDurableLookup(identifier) {
if (!identifier) return;
if (lookupTimerRef.current) window.clearTimeout(lookupTimerRef.current);
lookupTimerRef.current = window.setTimeout(() => {
performLookup(identifier, { silent: true }).catch(() => {});
}, QUOTE_LIFECYCLE_LOOKUP_DEBOUNCE_MS);
}
function inspectRow(item) {
const next = createSelectedLifecycleInvestigation(item, {
source: 'live_row',
now: Date.now(),
});
if (!next) return;
setSelectedInvestigation(next);
setLookupInput(next.identifier);
setLookupState('live');
setLookupError(null);
scheduleDurableLookup(next.identifier);
}
function submitLookup(event) {
event.preventDefault();
performLookup(lookupInput).catch(() => {});
}
function applyLatestLifecycleDisplay() { function applyLatestLifecycleDisplay() {
setDisplayItems(latestItemsRef.current || []); setDisplayItems(latestItemsRef.current || []);
setDisplayNow(Date.now()); setDisplayNow(Date.now());
@ -712,7 +1015,29 @@ function QuoteLifecycleTable({ items }) {
return ( return (
<> <>
<form className="quote-lifecycle-lookup" onSubmit={submitLookup}>
<label className="field quote-lifecycle-lookup-field">
<span>Lookup identifier</span>
<input
onChange={(event) => setLookupInput(event.target.value)}
placeholder="quote_id, decision_id, or command_id"
type="search"
value={lookupInput}
/>
</label>
<button className="button" disabled={!lookupInput.trim() || lookupState === 'loading'} type="submit">
Lookup
</button>
</form>
<div className="quote-lifecycle-controls"> <div className="quote-lifecycle-controls">
<label className="field quote-lifecycle-filter-field">
<span>Problem state</span>
<select value={problemFilter} onChange={(event) => setProblemFilter(event.target.value)}>
{QUOTE_LIFECYCLE_PROBLEM_FILTERS.map(([value, label]) => (
<option key={value} value={value}>{label}</option>
))}
</select>
</label>
<label className="toggle-field"> <label className="toggle-field">
<input <input
checked={showStrategyRejected} checked={showStrategyRejected}
@ -732,9 +1057,17 @@ function QuoteLifecycleTable({ items }) {
<Pill label={quoteDisplayPaused ? 'Display paused' : 'Live updates'} stateLabel={quoteDisplayPaused ? 'warning' : 'healthy'} /> <Pill label={quoteDisplayPaused ? 'Display paused' : 'Live updates'} stateLabel={quoteDisplayPaused ? 'warning' : 'healthy'} />
</div> </div>
<div className="status-subtle quote-lifecycle-snapshot-note"> <div className="status-subtle quote-lifecycle-snapshot-note">
Live rows update as they arrive. Fixed row slots and clamped cells keep the table height stable. Pause freezes only the rolling table. The pinned quote investigator can still refresh from durable lookup.
</div> </div>
<QuoteLifecycleInvestigator
lookupError={lookupError}
lookupState={lookupState}
now={liveNow}
onRefresh={(identifier) => performLookup(identifier).catch(() => {})}
selected={selectedInvestigation}
/>
<TableFrame className="quote-lifecycle-frame"> <TableFrame className="quote-lifecycle-frame">
<table className="quote-lifecycle-table"> <table className="quote-lifecycle-table">
<thead> <thead>
@ -760,13 +1093,18 @@ function QuoteLifecycleTable({ items }) {
} }
const rowKey = item.quote_id || item.decision_id || item.command_id || item.latest_stage_at || String(index); const rowKey = item.quote_id || item.decision_id || item.command_id || item.latest_stage_at || String(index);
const isExpanded = expanded.has(rowKey); const isExpanded = expanded.has(rowKey);
const isSelected = Boolean(
selectedInvestigation?.identifier
&& findLifecycleRowByIdentifier([item], selectedInvestigation.identifier),
);
const quoteTime = item.quote_activity_at || item.latest_stage_at; const quoteTime = item.quote_activity_at || item.latest_stage_at;
const updatedText = item.latest_stage_at && item.latest_stage_at !== item.quote_activity_at const updatedText = item.latest_stage_at && item.latest_stage_at !== item.quote_activity_at
? `Updated ${formatTimestamp(item.latest_stage_at)} - ${formatRelativeAge(item.latest_stage_at, displayNow)}` ? `Updated ${formatTimestamp(item.latest_stage_at)} - ${formatRelativeAge(item.latest_stage_at, displayNow)}`
: ''; : '';
const awaitingBrief = awaitingExecutorBrief(item, displayNow);
return ( return (
<Fragment key={rowKey}> <Fragment key={rowKey}>
<tr className={`quote-lifecycle-row${item.live_flash_at ? ' quote-row-flash' : ''}`} key={`${rowKey}:row`}> <tr className={`quote-lifecycle-row${item.live_flash_at ? ' quote-row-flash' : ''}${isSelected ? ' is-selected' : ''}`} key={`${rowKey}:row`}>
<td> <td>
<div className="quote-lifecycle-cell"> <div className="quote-lifecycle-cell">
<div className="lifecycle-line lifecycle-clamp-one">{formatTimestamp(quoteTime)}</div> <div className="lifecycle-line lifecycle-clamp-one">{formatTimestamp(quoteTime)}</div>
@ -788,8 +1126,9 @@ function QuoteLifecycleTable({ items }) {
<td><Pill label={item.lifecycle_label} stateLabel={item.lifecycle_tone} /></td> <td><Pill label={item.lifecycle_label} stateLabel={item.lifecycle_tone} /></td>
<td> <td>
<div className="quote-lifecycle-cell"> <div className="quote-lifecycle-cell">
<div className="lifecycle-line lifecycle-clamp-two">{item.reason_text}</div> <div className="lifecycle-line lifecycle-clamp-two">{item.reason_text || 'Unavailable'}</div>
<div className="status-subtle mono lifecycle-line lifecycle-clamp-one">{item.reason_code || 'reason_unknown'}</div> <div className="status-subtle mono lifecycle-line lifecycle-clamp-one">{item.reason_code || 'Unavailable'}</div>
{awaitingBrief ? <div className="status-subtle lifecycle-line lifecycle-clamp-two">{awaitingBrief}</div> : null}
</div> </div>
</td> </td>
<td> <td>
@ -800,14 +1139,19 @@ function QuoteLifecycleTable({ items }) {
</div> </div>
</td> </td>
<td> <td>
<button className="button secondary" onClick={() => toggle(rowKey)} type="button"> <div className="lifecycle-action-stack">
{isExpanded ? 'Hide lifecycle' : 'Show lifecycle'} <button className="button secondary" onClick={() => inspectRow(item)} type="button">
</button> Inspect quote
</button>
<button className="button secondary" onClick={() => toggle(rowKey)} type="button">
{isExpanded ? 'Hide lifecycle' : 'Show lifecycle'}
</button>
</div>
</td> </td>
</tr> </tr>
{isExpanded ? ( {isExpanded ? (
<tr className="lifecycle-expanded-row" key={`${rowKey}:details`}> <tr className="lifecycle-expanded-row" key={`${rowKey}:details`}>
<td colSpan={8}><LifecycleDetails item={item} /></td> <td colSpan={8}><LifecycleDetails item={item} now={displayNow} /></td>
</tr> </tr>
) : null} ) : null}
</Fragment> </Fragment>

View file

@ -618,10 +618,85 @@ table.lifecycle-table th:nth-child(5) {
margin-bottom: 12px; margin-bottom: 12px;
} }
.quote-lifecycle-lookup {
display: flex;
flex-wrap: wrap;
align-items: end;
gap: 10px;
margin-bottom: 12px;
}
.quote-lifecycle-lookup-field {
flex: 1 1 320px;
}
.quote-lifecycle-filter-field {
min-width: 230px;
}
.quote-lifecycle-lookup-field span,
.quote-lifecycle-filter-field span {
font-size: 0.84rem;
color: var(--muted);
}
.quote-lifecycle-snapshot-note { .quote-lifecycle-snapshot-note {
margin-bottom: 12px; margin-bottom: 12px;
} }
.quote-investigator-panel {
display: grid;
gap: 14px;
margin-bottom: 14px;
padding: 16px;
border: 1px solid var(--line);
border-radius: 16px;
background: rgba(255, 255, 255, 0.5);
}
.quote-investigator-head {
display: flex;
justify-content: space-between;
gap: 16px;
align-items: flex-start;
}
.quote-investigator-head h4 {
margin: 4px 0 0;
overflow-wrap: anywhere;
}
.quote-investigator-actions {
display: flex;
flex-wrap: wrap;
justify-content: flex-end;
gap: 8px;
}
.quote-investigator-facts {
display: grid;
gap: 12px;
grid-template-columns: repeat(auto-fit, minmax(190px, 1fr));
}
.quote-investigator-facts > div {
min-width: 0;
padding: 12px;
border: 1px solid var(--line);
border-radius: 14px;
background: var(--panel-2);
}
.quote-investigator-facts .status-value {
margin-top: 6px;
font-size: 1rem;
overflow-wrap: anywhere;
}
.quote-investigator-next {
line-height: 1.3;
}
.toggle-field { .toggle-field {
display: inline-flex; display: inline-flex;
align-items: center; align-items: center;
@ -651,6 +726,11 @@ table.lifecycle-table th:nth-child(5) {
height: 112px; height: 112px;
} }
.quote-lifecycle-table tbody tr.quote-lifecycle-row.is-selected td {
box-shadow: inset 3px 0 0 rgba(31, 122, 90, 0.76);
background: rgba(31, 122, 90, 0.08);
}
.quote-lifecycle-table td { .quote-lifecycle-table td {
overflow: hidden; overflow: hidden;
} }
@ -752,7 +832,12 @@ table.lifecycle-table th:nth-child(5) {
.quote-lifecycle-table td:nth-child(8), .quote-lifecycle-table td:nth-child(8),
.successful-trades-table th:nth-child(7), .successful-trades-table th:nth-child(7),
.successful-trades-table td:nth-child(7) { .successful-trades-table td:nth-child(7) {
width: 150px; width: 170px;
}
.lifecycle-action-stack {
display: grid;
gap: 8px;
} }
.successful-trades-table { .successful-trades-table {

View file

@ -37,6 +37,16 @@ test('operator dashboard API auth failures are JSON for frontend fetches', () =>
assert.match(source, /sendJson\(res, 401, \{ error: 'authentication_required' \}\)/); assert.match(source, /sendJson\(res, 401, \{ error: 'authentication_required' \}\)/);
}); });
test('operator dashboard exposes authenticated durable quote lifecycle lookup API', () => {
assert.match(source, /quote-lifecycle/);
assert.match(source, /lifecycleLookupMatch/);
assert.match(source, /loadQuoteLifecycleLookupEvidence/);
assert.match(source, /buildQuoteLifecycleLookupResponse/);
assert.match(source, /quote_lifecycle_identifier_required/);
assert.match(source, /quote_lifecycle_not_found/);
assert.match(source, /payload\.ok \? 200 : 404/);
});
test('operator dashboard bootstrap does not synchronously refresh intent request outcomes', () => { test('operator dashboard bootstrap does not synchronously refresh intent request outcomes', () => {
assert.match(source, /loadRecentIntentRequests\(pool, \{[\s\S]*refreshOutcomes: false/); assert.match(source, /loadRecentIntentRequests\(pool, \{[\s\S]*refreshOutcomes: false/);
}); });

View file

@ -32,9 +32,29 @@ test('strategy page owns consolidated quote lifecycle and successful trade table
assert.match(strategySource, /setDisplayItems\(items \|\| \[\]\)/); assert.match(strategySource, /setDisplayItems\(items \|\| \[\]\)/);
assert.match(strategySource, /useNow\(\)/); assert.match(strategySource, /useNow\(\)/);
assert.match(strategySource, /Live updates/); assert.match(strategySource, /Live updates/);
assert.match(strategySource, /Live rows update as they arrive/); assert.match(strategySource, /Pause freezes only the rolling table/);
assert.match(strategySource, /Pinned quote investigator/);
assert.match(strategySource, /Lookup identifier/);
assert.match(strategySource, /Inspect quote/);
assert.match(strategySource, /Refresh lookup/);
assert.match(strategySource, /fetchQuoteLifecycleLookup/);
assert.match(strategySource, /\/api\/strategy\/quote-lifecycle\//);
assert.match(strategySource, /QUOTE_LIFECYCLE_PROBLEM_FILTERS/);
assert.match(strategySource, /awaiting_executor/);
assert.match(strategySource, /quote_not_found_or_finished/);
assert.match(strategySource, /approved_no_command/);
assert.match(strategySource, /submitted_awaiting_outcome/);
assert.match(strategySource, /updateSelectedLifecycleFromLiveRows/);
assert.match(strategySource, /shouldRefreshSelectedFromDurable/);
assert.match(strategySource, /updateSelectedLifecycleFromLookup/);
assert.match(strategySource, /Command age:/);
assert.match(strategySource, /Executor result:/);
assert.match(strategySource, /Next expected:/);
assert.match(strategySource, /visibleRows/); assert.match(strategySource, /visibleRows/);
assert.match(strategySource, /quote-lifecycle-placeholder-row/); assert.match(strategySource, /quote-lifecycle-placeholder-row/);
assert.match(stylesSource, /\.quote-investigator-panel/);
assert.match(stylesSource, /\.quote-lifecycle-lookup/);
assert.match(stylesSource, /\.quote-lifecycle-table tbody tr\.quote-lifecycle-row\.is-selected/);
assert.match(stylesSource, /\.shell \{[\s\S]*?width: 100%;[\s\S]*?margin: 0;[\s\S]*?\}/); assert.match(stylesSource, /\.shell \{[\s\S]*?width: 100%;[\s\S]*?margin: 0;[\s\S]*?\}/);
assert.doesNotMatch(stylesSource, /^\s*max-width\s*:/m); assert.doesNotMatch(stylesSource, /^\s*max-width\s*:/m);
assert.doesNotMatch(strategySource, /maxWidth:/); assert.doesNotMatch(strategySource, /maxWidth:/);

View file

@ -7,6 +7,7 @@ import {
buildDashboardBootstrap, buildDashboardBootstrap,
buildDashboardControlErrorResponse, buildDashboardControlErrorResponse,
buildLiveStatusBar, buildLiveStatusBar,
buildQuoteLifecycleLookupResponse,
buildProfitabilitySummary, buildProfitabilitySummary,
createDashboardLiveState, createDashboardLiveState,
deriveQuoteLifecycleRows, deriveQuoteLifecycleRows,
@ -15,12 +16,19 @@ import {
resolveDashboardControlTimeoutMs, resolveDashboardControlTimeoutMs,
} from '../src/core/operator-dashboard.mjs'; } from '../src/core/operator-dashboard.mjs';
import { formatAge, formatAgeFromTimestamp } from '../src/operator-dashboard/static/lib/format.js'; import { formatAge, formatAgeFromTimestamp } from '../src/operator-dashboard/static/lib/format.js';
import {
createSelectedLifecycleInvestigation,
shouldRefreshSelectedFromDurable,
updateSelectedLifecycleFromLiveRows,
updateSelectedLifecycleFromLookup,
} from '../src/operator-dashboard/static/lib/quoteLifecycleInvestigator.js';
import { dashboardReducer } from '../src/operator-dashboard/static/state/dashboardReducer.js'; import { dashboardReducer } from '../src/operator-dashboard/static/state/dashboardReducer.js';
import { import {
buildDashboardSessionToken, buildDashboardSessionToken,
parseBasicAuthorizationHeader, parseBasicAuthorizationHeader,
resolveDashboardRequestAuth, resolveDashboardRequestAuth,
} from '../src/core/operator-dashboard-auth.mjs'; } from '../src/core/operator-dashboard-auth.mjs';
import { loadQuoteLifecycleLookupEvidence } from '../src/lib/postgres.mjs';
function buildConfig() { function buildConfig() {
const tradingBtc = { const tradingBtc = {
@ -564,6 +572,272 @@ test('socket lifecycle messages replace strategy rows without page refresh', ()
assert.equal(next.dashboard.strategy.strategy_state.recent_lifecycle_rows[0].live_flash_at, '2026-04-04T09:00:00.000Z'); assert.equal(next.dashboard.strategy.strategy_state.recent_lifecycle_rows[0].live_flash_at, '2026-04-04T09:00:00.000Z');
}); });
test('selected awaiting executor quote remains inspectable after live rows move and refreshes from durable result', () => {
const config = buildConfig();
const awaiting = buildQuoteLifecycleLookupResponse({
config,
identifier: 'quote-follow',
evidence: {
lookup_at: '2026-06-12T10:00:00.000Z',
matched_identifier_type: 'quote_id',
matched_identifiers: {
quote_ids: ['quote-follow'],
decision_ids: ['decision-follow'],
command_ids: ['cmd-follow'],
},
recentTradeDecisions: [{
observed_at: '2026-06-12T09:59:58.000Z',
payload: {
quote_id: 'quote-follow',
decision_id: 'decision-follow',
pair: config.activePair,
decision: 'actionable',
decision_reason: 'actionable',
},
}],
recentExecuteTradeCommands: [{
observed_at: '2026-06-12T09:59:59.000Z',
command_id: 'cmd-follow',
decision_id: 'decision-follow',
quote_id: 'quote-follow',
pair: config.activePair,
}],
},
});
assert.equal(awaiting.ok, true);
assert.equal(awaiting.lifecycle_row.lifecycle_state, 'command_emitted');
assert.equal(awaiting.lifecycle_row.lifecycle_label, 'Awaiting executor');
const selected = createSelectedLifecycleInvestigation(awaiting.lifecycle_row, {
now: '2026-06-12T10:00:00.000Z',
});
const afterLiveMove = updateSelectedLifecycleFromLiveRows(selected, [{
quote_id: 'quote-newer',
lifecycle_state: 'observed',
}], {
now: '2026-06-12T10:00:02.000Z',
});
assert.equal(afterLiveMove.row.quote_id, 'quote-follow');
assert.equal(afterLiveMove.row.lifecycle_state, 'command_emitted');
assert.equal(shouldRefreshSelectedFromDurable(afterLiveMove, [{ quote_id: 'quote-newer' }]), true);
const later = buildQuoteLifecycleLookupResponse({
config,
identifier: 'cmd-follow',
evidence: {
lookup_at: '2026-06-12T10:00:04.000Z',
matched_identifier_type: 'command_id',
matched_identifiers: {
quote_ids: ['quote-follow'],
decision_ids: ['decision-follow'],
command_ids: ['cmd-follow'],
},
recentExecuteTradeCommands: [{
observed_at: '2026-06-12T09:59:59.000Z',
command_id: 'cmd-follow',
decision_id: 'decision-follow',
quote_id: 'quote-follow',
pair: config.activePair,
}],
recentExecutionResults: [{
command_id: 'cmd-follow',
decision_id: 'decision-follow',
quote_id: 'quote-follow',
pair: config.activePair,
result_at: '2026-06-12T10:00:03.000Z',
status: 'failed',
result_code: 'submission_failed',
failure_category: 'quote_not_found_or_finished',
note: 'relay no longer has the quote',
}],
},
});
const refreshed = updateSelectedLifecycleFromLookup(afterLiveMove, later, {
identifier: 'cmd-follow',
now: '2026-06-12T10:00:04.000Z',
});
assert.equal(refreshed.source, 'durable_lookup');
assert.equal(refreshed.row.quote_id, 'quote-follow');
assert.equal(refreshed.row.lifecycle_state, 'failed');
assert.equal(refreshed.row.reason_code, 'quote_not_found_or_finished');
assert.match(refreshed.row.reason_text, /not found or already finished/i);
});
test('quote lifecycle lookup response matches quote, decision, and command identifiers from durable evidence', () => {
const config = buildConfig();
const evidence = {
lookup_at: '2026-06-12T10:10:00.000Z',
matched_identifiers: {
quote_ids: ['quote-lookup'],
decision_ids: ['decision-lookup'],
command_ids: ['cmd-lookup'],
},
recentQuotes: [{
quote_id: 'quote-lookup',
pair: config.activePair,
asset_in: config.tradingBtc.assetId,
asset_out: config.tradingEure.assetId,
amount_in: '100',
amount_out: '200',
observed_at: '2026-06-12T10:09:57.000Z',
}],
recentTradeDecisions: [{
observed_at: '2026-06-12T10:09:58.000Z',
payload: {
quote_id: 'quote-lookup',
decision_id: 'decision-lookup',
pair: config.activePair,
decision: 'actionable',
decision_reason: 'actionable',
},
}],
recentExecuteTradeCommands: [{
observed_at: '2026-06-12T10:09:59.000Z',
command_id: 'cmd-lookup',
decision_id: 'decision-lookup',
quote_id: 'quote-lookup',
pair: config.activePair,
}],
};
const byQuote = buildQuoteLifecycleLookupResponse({
config,
identifier: 'quote-lookup',
evidence: { ...evidence, matched_identifier_type: 'quote_id' },
});
const byDecision = buildQuoteLifecycleLookupResponse({
config,
identifier: 'decision-lookup',
evidence: { ...evidence, matched_identifier_type: 'decision_id' },
});
const byCommand = buildQuoteLifecycleLookupResponse({
config,
identifier: 'cmd-lookup',
evidence: { ...evidence, matched_identifier_type: 'command_id' },
});
assert.equal(byQuote.lifecycle_row.quote_id, 'quote-lookup');
assert.equal(byDecision.lifecycle_row.decision_id, 'decision-lookup');
assert.equal(byCommand.lifecycle_row.command_id, 'cmd-lookup');
assert.equal(byQuote.lookup.matched_identifier_type, 'quote_id');
assert.equal(byDecision.lookup.matched_identifier_type, 'decision_id');
assert.equal(byCommand.lookup.matched_identifier_type, 'command_id');
});
test('postgres quote lifecycle lookup loader links durable quote, decision, and command evidence', async () => {
const queries = [];
const pool = {
async query(sql, params = []) {
queries.push({ sql, params });
if (sql.includes('WITH candidates')) {
return {
rows: [{
matched_identifier_type: 'decision_id',
quote_id: 'quote-loader',
decision_id: 'decision-loader',
command_id: 'cmd-loader',
matched_at: '2026-06-12T10:20:00.000Z',
}],
};
}
if (sql.includes('FROM swap_demand_events')) {
return {
rows: [{
observed_at: '2026-06-12T10:19:57.000Z',
ingested_at: '2026-06-12T10:19:57.000Z',
payload: {
quote_id: 'quote-loader',
pair: 'btc->eure',
asset_in: 'btc',
asset_out: 'eure',
amount_in: '100',
amount_out: '200',
},
}],
};
}
if (sql.includes('FROM trade_decisions')) {
return {
rows: [{
observed_at: '2026-06-12T10:19:58.000Z',
ingested_at: '2026-06-12T10:19:58.000Z',
payload: {
quote_id: 'quote-loader',
decision_id: 'decision-loader',
decision: 'actionable',
decision_reason: 'actionable',
},
}],
};
}
if (sql.includes('FROM execute_trade_commands')) {
return {
rows: [{
observed_at: '2026-06-12T10:19:59.000Z',
ingested_at: '2026-06-12T10:19:59.000Z',
payload: {
quote_id: 'quote-loader',
decision_id: 'decision-loader',
command_id: 'cmd-loader',
},
}],
};
}
if (sql.includes('FROM trade_execution_results r')) {
return {
rows: [{
result_observed_at: '2026-06-12T10:20:01.000Z',
result_ingested_at: '2026-06-12T10:20:01.000Z',
result_payload: {
quote_id: 'quote-loader',
decision_id: 'decision-loader',
command_id: 'cmd-loader',
status: 'failed',
result_code: 'submission_failed',
failure_category: 'quote_not_found_or_finished',
},
command_ingested_at: '2026-06-12T10:19:59.000Z',
command_payload: {
quote_id: 'quote-loader',
decision_id: 'decision-loader',
command_id: 'cmd-loader',
},
decision_payload: {
quote_id: 'quote-loader',
decision_id: 'decision-loader',
},
outcome_payload: null,
}],
};
}
if (sql.includes('FROM quote_outcome_attributions')) {
return { rows: [] };
}
return { rows: [] };
},
};
const evidence = await loadQuoteLifecycleLookupEvidence(pool, {
identifier: 'decision-loader',
});
assert.equal(evidence.found, true);
assert.equal(evidence.matched_identifier_type, 'decision_id');
assert.deepEqual(evidence.matched_identifiers, {
quote_ids: ['quote-loader'],
decision_ids: ['decision-loader'],
command_ids: ['cmd-loader'],
});
assert.equal(evidence.recentQuotes[0].quote_id, 'quote-loader');
assert.equal(evidence.recentTradeDecisions[0].payload.decision_id, 'decision-loader');
assert.equal(evidence.recentExecuteTradeCommands[0].command_id, 'cmd-loader');
assert.equal(evidence.recentExecutionResults[0].failure_category, 'quote_not_found_or_finished');
assert.equal(queries[0].params[0], 'decision-loader');
});
test('dashboard age formatting uses seconds first and exact timestamp deltas', () => { test('dashboard age formatting uses seconds first and exact timestamp deltas', () => {
assert.equal(formatAge(999), '999 ms'); assert.equal(formatAge(999), '999 ms');
assert.equal(formatAge(1_500), '1s'); assert.equal(formatAge(1_500), '1s');