unrip/IMPLEMENTATION.md

# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails

Status: open
Opened: 2026-06-12

## Goal
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.

## Selected backlog items
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.

## Design rules
- This turn is execution latency and visibility only.
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
- Do not skip quotes because relay errors are likely.
- Do not submit a quote response if signing prerequisites are unavailable.
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
- Keep old executor result rows readable.
- Deploy only through repo workflow.

## Problem statement
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing.

Live evidence showed this can block for seconds:

- `ae824f64-ddfd-4310-a44f-69c401fc8ad7`
  - `current_salt_source=refresh`
  - `current_salt_ms=7481.458`
  - executor received the command quickly, then blocked inside salt refresh
- following commands waited in the single executor consumer queue:
  - `command_to_executor_ms` around `7.5s`
  - some later rows around `10-12s`
  - most failed with `quote_not_found_or_finished`

External verifier latency must not sit on the per-quote execution critical path.

## Backend changes

### 1. Inspect current salt and executor paths
- Inspect:
  - `src/core/verifier-salt-cache.mjs`
  - `src/apps/trade-executor.mjs`
  - executor command expiry helpers
  - executor timing payload construction
  - runtime health and service snapshot summaries
  - ops sentinel alert generation
  - operator dashboard service/system views
  - existing verifier salt cache and executor tests
- Identify every call path that can refresh verifier salt.
- Confirm which call paths are quote-response hot path versus background or operator-triggered work.

### 2. Define salt cache contract
- Add or refine a cache API with separate responsibilities:
  - background refresh may perform network I/O
  - hot-path read must never perform network I/O
- Candidate API:
  - `start()` starts periodic refresh and immediate refresh attempt
  - `refreshNow()` performs explicit refresh with in-flight de-duplication
  - `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O
  - `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
- Validate malformed salt before it can enter the cache.

### 3. Background refresh behavior
- Ensure startup kicks off refresh without making the executor command handler wait on it.
- Refresh periodically before the salt becomes stale.
- If refresh fails:
  - keep a still-fresh cached salt usable until its age limit expires
  - expose last error and stale/unavailable state
  - do not block command handling
- Record:
  - `last_refresh_started_at`
  - `last_refresh_completed_at`
  - `last_refresh_duration_ms`
  - `last_refresh_error`
  - `refresh_in_flight`
  - `salt_age_ms`
  - `salt_fresh`

### 4. Trade-executor hot path
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read.
- If no fresh cached salt is available:
  - publish a durable trade execution result
  - use status `rejected`
  - use explicit result code such as `verifier_salt_unavailable`
  - include salt cache state summary in the result payload
  - do not call `buildQuoteResponseSubmission`
  - do not call the relay
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`.
- Keep idempotency and executor state-store semantics intact.
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.

### 5. Queue delay guardrails
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`.
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
- Add thresholded runtime health fields:
  - recent max command-to-executor
  - p90 or bucketed queue delay if practical
  - latest command queue delay warning reason
- Do not use queue delay to change strategy selection or pricing in this turn.

### 6. Runtime health and alerts
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status.
- Add ops-sentinel warning alerts for:
  - verifier salt stale or unavailable
  - verifier salt refresh failing
  - executor command queue delay above a conservative threshold
- Keep alert scopes executor/service-level unless pair-specific evidence is available.
- Do not disarm automatically unless an existing safety invariant already does so.

### 7. Operator dashboard
- Surface salt cache state in the system/service view:
  - fresh/stale/unavailable
  - salt age
  - last refresh time
  - last refresh duration
  - last refresh error
- Surface executor queue delay alongside existing trade-executor health.
- Preserve existing lifecycle table timing fields and old-row compatibility.
- Avoid presenting salt failures as trade PnL or settled outcomes.

### 8. Tests
- Add verifier salt cache tests:
  - cached fresh read returns immediately and does not call the loader
  - missing/stale salt returns unavailable without loader call
  - background refresh records success duration and freshness
  - background refresh records error without destroying still-fresh salt
  - malformed salts fail closed
- Add trade-executor tests or refactorable helper tests:
  - slow salt loader cannot block quote command handling
  - no cached fresh salt publishes `verifier_salt_unavailable`
  - no relay call happens when salt is unavailable
  - stale command expiry still happens before salt lookup
  - successful command uses cached salt and records `current_salt_source=cache`
- Add runtime health and alert tests for salt and queue warnings.
- Add dashboard/static tests for salt and queue visibility.

### 9. Validation and deploy
- Run targeted tests:
  - verifier salt cache
  - trade executor
  - runtime health / ops sentinel
  - dashboard static/operator tests
- Run full `npm test`.
- Build operator dashboard bundle.
- Commit with required workflow metadata.
- Push to `main` so repo workflow deploys.
- Do not manually patch or roll Kubernetes deployments.

### 10. Live evidence after deploy
- Confirm all repo-owned deployments run the new image.
- Query recent trade execution results:
  - `current_salt_source`
  - `current_salt_ms`
  - `command_to_executor_ms`
  - `quote_age_at_executor_receipt_ms`
  - `result_code`
  - `failure_category`
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
- Report command-to-executor p50/p90/p99 after rollout.
- Report salt cache health from trade-executor state or health.
- Confirm recent quote flow still writes decisions, commands, and results.

## Data and persistence
- Use existing executor result payloads for per-command salt and queue timing.
- Add fields additively so old rows remain readable.
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
- Do not add new long-term storage unless required for dashboard or alert correctness.

## Edge cases
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
- Salt cache stale: commands reject quickly, no relay call.
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
- Malformed salt from verifier: fail closed and expose error.
- Existing stale command: reject as stale command before salt lookup.
- Executor paused or disarmed: existing behavior remains.
- Old result rows without salt fields remain visible.

## Concrete implementation order

### Phase 1. Audit and contracts
- Inspect current salt cache and executor usage.
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
- Decide exact result code and payload shape for salt-unavailable rejection.

### Phase 2. Salt cache refactor
- Implement non-blocking cached salt read.
- Implement refresh state fields and in-flight de-duplication.
- Ensure background refresh starts and continues.
- Keep existing cache tests passing with updated contract.

### Phase 3. Executor integration
- Change quote-response hot path to use cached salt only.
- Add salt-unavailable rejection result.
- Add timing/state fields for cache read and unavailable rejection.
- Add regression tests for slow salt loader and no relay call.

### Phase 4. Health, alerts, dashboard
- Expose salt freshness and queue delay in trade-executor state/health.
- Add ops-sentinel/runtime alert tests.
- Add dashboard visibility tests and UI fields.

### Phase 5. Validation and deployment
- Run targeted tests.
- Run full `npm test`.
- Build dashboard bundle.
- Commit and push.
- Collect live post-deploy evidence.

## Test plan
- `test/verifier-salt-cache.test.mjs`
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
- `test/runtime-health.test.mjs`
- `test/ops-sentinel*.test.mjs`
- `test/operator-dashboard*.test.mjs`
- Static config tests proving no new active env knobs are added
- Full `npm test`
- Dashboard build

## Validation checklist against the proof
- Quote-response hot path contains no verifier salt network refresh.
- Slow `current_salt` loader does not cause multi-second command queue delay.
- Missing/stale salt creates durable explicit rejection without relay call.
- Salt freshness and refresh errors are operator-visible.
- Executor queue delay is operator-visible and alertable.
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
- Repo workflow deploys the result.
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.

## Known fakes allowed at start of this turn
- Venue-native terminal fill ids remain unavailable.
- Fee-complete realized PnL remains unavailable.
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.