unrip/archive/implementation/20260612T164553Z-verifier-salt-hot-path-removal-and-executor-queue-guardrails-implementation.md

Implementation Turn: Verifier salt hot-path removal and executor queue guardrails

Status: open Opened: 2026-06-12

Goal

Remove verifier current_salt RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.

Selected backlog items

• none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.

Design rules

• This turn is execution latency and visibility only.
• Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
• Do not skip quotes because relay errors are likely.
• Do not submit a quote response if signing prerequisites are unavailable.
• Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
• Keep old executor result rows readable.
• Deploy only through repo workflow.

Problem statement

The quote-response executor currently calls verifierSaltCache.getFreshSalt() from inside handleCommand. If the cache decides it needs to refresh, the command waits for the NEAR verifier current_salt RPC before signing.

Live evidence showed this can block for seconds:

• ae824f64-ddfd-4310-a44f-69c401fc8ad7
  • current_salt_source=refresh
  • current_salt_ms=7481.458
  • executor received the command quickly, then blocked inside salt refresh
• following commands waited in the single executor consumer queue:
  • command_to_executor_ms around 7.5s
  • some later rows around 10-12s
  • most failed with quote_not_found_or_finished

External verifier latency must not sit on the per-quote execution critical path.

Backend changes

1. Inspect current salt and executor paths

• Inspect:
  • src/core/verifier-salt-cache.mjs
  • src/apps/trade-executor.mjs
  • executor command expiry helpers
  • executor timing payload construction
  • runtime health and service snapshot summaries
  • ops sentinel alert generation
  • operator dashboard service/system views
  • existing verifier salt cache and executor tests
• Identify every call path that can refresh verifier salt.
• Confirm which call paths are quote-response hot path versus background or operator-triggered work.

2. Define salt cache contract

• Add or refine a cache API with separate responsibilities:
  • background refresh may perform network I/O
  • hot-path read must never perform network I/O
• Candidate API:
  • start() starts periodic refresh and immediate refresh attempt
  • refreshNow() performs explicit refresh with in-flight de-duplication
  • getCachedFreshSalt({ now, maxAgeMs }) returns fresh salt or a typed unavailable result without I/O
  • getState() returns freshness, age, last refresh times, duration, in-flight state, and error
• Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
• Validate malformed salt before it can enter the cache.

3. Background refresh behavior

• Ensure startup kicks off refresh without making the executor command handler wait on it.
• Refresh periodically before the salt becomes stale.
• If refresh fails:
  • keep a still-fresh cached salt usable until its age limit expires
  • expose last error and stale/unavailable state
  • do not block command handling
• Record:
  • last_refresh_started_at
  • last_refresh_completed_at
  • last_refresh_duration_ms
  • last_refresh_error
  • refresh_in_flight
  • salt_age_ms
  • salt_fresh

4. Trade-executor hot path

• Replace hot-path await verifierSaltCache.getFreshSalt() with non-blocking cached salt read.
• If no fresh cached salt is available:
  • publish a durable trade execution result
  • use status rejected
  • use explicit result code such as verifier_salt_unavailable
  • include salt cache state summary in the result payload
  • do not call buildQuoteResponseSubmission
  • do not call the relay
• Keep command expiry before signing so stale queued commands remain rejected as stale_execute_command.
• Keep idempotency and executor state-store semantics intact.
• Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.

5. Queue delay guardrails

• Keep recording command_to_executor_ms and quote_age_at_executor_receipt_ms.
• Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
• Add thresholded runtime health fields:
  • recent max command-to-executor
  • p90 or bucketed queue delay if practical
  • latest command queue delay warning reason
• Do not use queue delay to change strategy selection or pricing in this turn.

6. Runtime health and alerts

• Extend trade-executor /state and /health output with salt cache freshness and queue-delay status.
• Add ops-sentinel warning alerts for:
  • verifier salt stale or unavailable
  • verifier salt refresh failing
  • executor command queue delay above a conservative threshold
• Keep alert scopes executor/service-level unless pair-specific evidence is available.
• Do not disarm automatically unless an existing safety invariant already does so.

7. Operator dashboard

• Surface salt cache state in the system/service view:
  • fresh/stale/unavailable
  • salt age
  • last refresh time
  • last refresh duration
  • last refresh error
• Surface executor queue delay alongside existing trade-executor health.
• Preserve existing lifecycle table timing fields and old-row compatibility.
• Avoid presenting salt failures as trade PnL or settled outcomes.

8. Tests

• Add verifier salt cache tests:
  • cached fresh read returns immediately and does not call the loader
  • missing/stale salt returns unavailable without loader call
  • background refresh records success duration and freshness
  • background refresh records error without destroying still-fresh salt
  • malformed salts fail closed
• Add trade-executor tests or refactorable helper tests:
  • slow salt loader cannot block quote command handling
  • no cached fresh salt publishes verifier_salt_unavailable
  • no relay call happens when salt is unavailable
  • stale command expiry still happens before salt lookup
  • successful command uses cached salt and records current_salt_source=cache
• Add runtime health and alert tests for salt and queue warnings.
• Add dashboard/static tests for salt and queue visibility.

9. Validation and deploy

• Run targeted tests:
  • verifier salt cache
  • trade executor
  • runtime health / ops sentinel
  • dashboard static/operator tests
• Run full npm test.
• Build operator dashboard bundle.
• Commit with required workflow metadata.
• Push to main so repo workflow deploys.
• Do not manually patch or roll Kubernetes deployments.

10. Live evidence after deploy

• Confirm all repo-owned deployments run the new image.
• Query recent trade execution results:
  • current_salt_source
  • current_salt_ms
  • command_to_executor_ms
  • quote_age_at_executor_receipt_ms
  • result_code
  • failure_category
• Prove no post-deploy quote-response result used current_salt_source=refresh.
• Report command-to-executor p50/p90/p99 after rollout.
• Report salt cache health from trade-executor state or health.
• Confirm recent quote flow still writes decisions, commands, and results.

Data and persistence

• Use existing executor result payloads for per-command salt and queue timing.
• Add fields additively so old rows remain readable.
• Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
• Do not add new long-term storage unless required for dashboard or alert correctness.

Edge cases

• Salt cache empty at startup: commands reject quickly until background refresh succeeds.
• Salt cache stale: commands reject quickly, no relay call.
• Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
• Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
• Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
• Malformed salt from verifier: fail closed and expose error.
• Existing stale command: reject as stale command before salt lookup.
• Executor paused or disarmed: existing behavior remains.
• Old result rows without salt fields remain visible.

Concrete implementation order

Phase 1. Audit and contracts

• Inspect current salt cache and executor usage.
• Add salt-cache contract tests for non-blocking cached reads and refresh state.
• Decide exact result code and payload shape for salt-unavailable rejection.

Phase 2. Salt cache refactor

• Implement non-blocking cached salt read.
• Implement refresh state fields and in-flight de-duplication.
• Ensure background refresh starts and continues.
• Keep existing cache tests passing with updated contract.

Phase 3. Executor integration

• Change quote-response hot path to use cached salt only.
• Add salt-unavailable rejection result.
• Add timing/state fields for cache read and unavailable rejection.
• Add regression tests for slow salt loader and no relay call.

Phase 4. Health, alerts, dashboard

• Expose salt freshness and queue delay in trade-executor state/health.
• Add ops-sentinel/runtime alert tests.
• Add dashboard visibility tests and UI fields.

Phase 5. Validation and deployment

• Run targeted tests.
• Run full npm test.
• Build dashboard bundle.
• Commit and push.
• Collect live post-deploy evidence.

Test plan

• test/verifier-salt-cache.test.mjs
• test/trade-executor*.test.mjs or new extracted executor helper tests
• test/runtime-health.test.mjs
• test/ops-sentinel*.test.mjs
• test/operator-dashboard*.test.mjs
• Static config tests proving no new active env knobs are added
• Full npm test
• Dashboard build

Validation checklist against the proof

• Quote-response hot path contains no verifier salt network refresh.
• Slow current_salt loader does not cause multi-second command queue delay.
• Missing/stale salt creates durable explicit rejection without relay call.
• Salt freshness and refresh errors are operator-visible.
• Executor queue delay is operator-visible and alertable.
• Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
• Repo workflow deploys the result.
• Live results show current_salt_source=cache or salt-unavailable rejection, not hot-path refresh.

Known fakes allowed at start of this turn

• Venue-native terminal fill ids remain unavailable.
• Fee-complete realized PnL remains unavailable.
• Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.