cf90597678
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive. Assumptions: The archived files capture the relevant planning state for the completed turn. Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
235 lines
11 KiB
Markdown
235 lines
11 KiB
Markdown
# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails
|
|
|
|
Status: open
|
|
Opened: 2026-06-12
|
|
|
|
## Goal
|
|
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.
|
|
|
|
## Selected backlog items
|
|
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.
|
|
|
|
## Design rules
|
|
- This turn is execution latency and visibility only.
|
|
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
|
|
- Do not skip quotes because relay errors are likely.
|
|
- Do not submit a quote response if signing prerequisites are unavailable.
|
|
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
|
- Keep old executor result rows readable.
|
|
- Deploy only through repo workflow.
|
|
|
|
## Problem statement
|
|
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing.
|
|
|
|
Live evidence showed this can block for seconds:
|
|
|
|
- `ae824f64-ddfd-4310-a44f-69c401fc8ad7`
|
|
- `current_salt_source=refresh`
|
|
- `current_salt_ms=7481.458`
|
|
- executor received the command quickly, then blocked inside salt refresh
|
|
- following commands waited in the single executor consumer queue:
|
|
- `command_to_executor_ms` around `7.5s`
|
|
- some later rows around `10-12s`
|
|
- most failed with `quote_not_found_or_finished`
|
|
|
|
External verifier latency must not sit on the per-quote execution critical path.
|
|
|
|
## Backend changes
|
|
|
|
### 1. Inspect current salt and executor paths
|
|
- Inspect:
|
|
- `src/core/verifier-salt-cache.mjs`
|
|
- `src/apps/trade-executor.mjs`
|
|
- executor command expiry helpers
|
|
- executor timing payload construction
|
|
- runtime health and service snapshot summaries
|
|
- ops sentinel alert generation
|
|
- operator dashboard service/system views
|
|
- existing verifier salt cache and executor tests
|
|
- Identify every call path that can refresh verifier salt.
|
|
- Confirm which call paths are quote-response hot path versus background or operator-triggered work.
|
|
|
|
### 2. Define salt cache contract
|
|
- Add or refine a cache API with separate responsibilities:
|
|
- background refresh may perform network I/O
|
|
- hot-path read must never perform network I/O
|
|
- Candidate API:
|
|
- `start()` starts periodic refresh and immediate refresh attempt
|
|
- `refreshNow()` performs explicit refresh with in-flight de-duplication
|
|
- `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O
|
|
- `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error
|
|
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
|
|
- Validate malformed salt before it can enter the cache.
|
|
|
|
### 3. Background refresh behavior
|
|
- Ensure startup kicks off refresh without making the executor command handler wait on it.
|
|
- Refresh periodically before the salt becomes stale.
|
|
- If refresh fails:
|
|
- keep a still-fresh cached salt usable until its age limit expires
|
|
- expose last error and stale/unavailable state
|
|
- do not block command handling
|
|
- Record:
|
|
- `last_refresh_started_at`
|
|
- `last_refresh_completed_at`
|
|
- `last_refresh_duration_ms`
|
|
- `last_refresh_error`
|
|
- `refresh_in_flight`
|
|
- `salt_age_ms`
|
|
- `salt_fresh`
|
|
|
|
### 4. Trade-executor hot path
|
|
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read.
|
|
- If no fresh cached salt is available:
|
|
- publish a durable trade execution result
|
|
- use status `rejected`
|
|
- use explicit result code such as `verifier_salt_unavailable`
|
|
- include salt cache state summary in the result payload
|
|
- do not call `buildQuoteResponseSubmission`
|
|
- do not call the relay
|
|
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`.
|
|
- Keep idempotency and executor state-store semantics intact.
|
|
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.
|
|
|
|
### 5. Queue delay guardrails
|
|
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`.
|
|
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
|
|
- Add thresholded runtime health fields:
|
|
- recent max command-to-executor
|
|
- p90 or bucketed queue delay if practical
|
|
- latest command queue delay warning reason
|
|
- Do not use queue delay to change strategy selection or pricing in this turn.
|
|
|
|
### 6. Runtime health and alerts
|
|
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status.
|
|
- Add ops-sentinel warning alerts for:
|
|
- verifier salt stale or unavailable
|
|
- verifier salt refresh failing
|
|
- executor command queue delay above a conservative threshold
|
|
- Keep alert scopes executor/service-level unless pair-specific evidence is available.
|
|
- Do not disarm automatically unless an existing safety invariant already does so.
|
|
|
|
### 7. Operator dashboard
|
|
- Surface salt cache state in the system/service view:
|
|
- fresh/stale/unavailable
|
|
- salt age
|
|
- last refresh time
|
|
- last refresh duration
|
|
- last refresh error
|
|
- Surface executor queue delay alongside existing trade-executor health.
|
|
- Preserve existing lifecycle table timing fields and old-row compatibility.
|
|
- Avoid presenting salt failures as trade PnL or settled outcomes.
|
|
|
|
### 8. Tests
|
|
- Add verifier salt cache tests:
|
|
- cached fresh read returns immediately and does not call the loader
|
|
- missing/stale salt returns unavailable without loader call
|
|
- background refresh records success duration and freshness
|
|
- background refresh records error without destroying still-fresh salt
|
|
- malformed salts fail closed
|
|
- Add trade-executor tests or refactorable helper tests:
|
|
- slow salt loader cannot block quote command handling
|
|
- no cached fresh salt publishes `verifier_salt_unavailable`
|
|
- no relay call happens when salt is unavailable
|
|
- stale command expiry still happens before salt lookup
|
|
- successful command uses cached salt and records `current_salt_source=cache`
|
|
- Add runtime health and alert tests for salt and queue warnings.
|
|
- Add dashboard/static tests for salt and queue visibility.
|
|
|
|
### 9. Validation and deploy
|
|
- Run targeted tests:
|
|
- verifier salt cache
|
|
- trade executor
|
|
- runtime health / ops sentinel
|
|
- dashboard static/operator tests
|
|
- Run full `npm test`.
|
|
- Build operator dashboard bundle.
|
|
- Commit with required workflow metadata.
|
|
- Push to `main` so repo workflow deploys.
|
|
- Do not manually patch or roll Kubernetes deployments.
|
|
|
|
### 10. Live evidence after deploy
|
|
- Confirm all repo-owned deployments run the new image.
|
|
- Query recent trade execution results:
|
|
- `current_salt_source`
|
|
- `current_salt_ms`
|
|
- `command_to_executor_ms`
|
|
- `quote_age_at_executor_receipt_ms`
|
|
- `result_code`
|
|
- `failure_category`
|
|
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
|
|
- Report command-to-executor p50/p90/p99 after rollout.
|
|
- Report salt cache health from trade-executor state or health.
|
|
- Confirm recent quote flow still writes decisions, commands, and results.
|
|
|
|
## Data and persistence
|
|
- Use existing executor result payloads for per-command salt and queue timing.
|
|
- Add fields additively so old rows remain readable.
|
|
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
|
|
- Do not add new long-term storage unless required for dashboard or alert correctness.
|
|
|
|
## Edge cases
|
|
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
|
|
- Salt cache stale: commands reject quickly, no relay call.
|
|
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
|
|
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
|
|
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
|
|
- Malformed salt from verifier: fail closed and expose error.
|
|
- Existing stale command: reject as stale command before salt lookup.
|
|
- Executor paused or disarmed: existing behavior remains.
|
|
- Old result rows without salt fields remain visible.
|
|
|
|
## Concrete implementation order
|
|
|
|
### Phase 1. Audit and contracts
|
|
- Inspect current salt cache and executor usage.
|
|
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
|
|
- Decide exact result code and payload shape for salt-unavailable rejection.
|
|
|
|
### Phase 2. Salt cache refactor
|
|
- Implement non-blocking cached salt read.
|
|
- Implement refresh state fields and in-flight de-duplication.
|
|
- Ensure background refresh starts and continues.
|
|
- Keep existing cache tests passing with updated contract.
|
|
|
|
### Phase 3. Executor integration
|
|
- Change quote-response hot path to use cached salt only.
|
|
- Add salt-unavailable rejection result.
|
|
- Add timing/state fields for cache read and unavailable rejection.
|
|
- Add regression tests for slow salt loader and no relay call.
|
|
|
|
### Phase 4. Health, alerts, dashboard
|
|
- Expose salt freshness and queue delay in trade-executor state/health.
|
|
- Add ops-sentinel/runtime alert tests.
|
|
- Add dashboard visibility tests and UI fields.
|
|
|
|
### Phase 5. Validation and deployment
|
|
- Run targeted tests.
|
|
- Run full `npm test`.
|
|
- Build dashboard bundle.
|
|
- Commit and push.
|
|
- Collect live post-deploy evidence.
|
|
|
|
## Test plan
|
|
- `test/verifier-salt-cache.test.mjs`
|
|
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
|
|
- `test/runtime-health.test.mjs`
|
|
- `test/ops-sentinel*.test.mjs`
|
|
- `test/operator-dashboard*.test.mjs`
|
|
- Static config tests proving no new active env knobs are added
|
|
- Full `npm test`
|
|
- Dashboard build
|
|
|
|
## Validation checklist against the proof
|
|
- Quote-response hot path contains no verifier salt network refresh.
|
|
- Slow `current_salt` loader does not cause multi-second command queue delay.
|
|
- Missing/stale salt creates durable explicit rejection without relay call.
|
|
- Salt freshness and refresh errors are operator-visible.
|
|
- Executor queue delay is operator-visible and alertable.
|
|
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
|
|
- Repo workflow deploys the result.
|
|
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.
|
|
|
|
## Known fakes allowed at start of this turn
|
|
- Venue-native terminal fill ids remain unavailable.
|
|
- Fee-complete realized PnL remains unavailable.
|
|
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.
|