Archive implementation turn: Verifier salt hot-path removal and executor queue guardrails
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive. Assumptions: The archived files capture the relevant planning state for the completed turn. Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
This commit is contained in:
philipp
parent
ddd3dfb9e2
commit
cf90597678
5 changed files with 386 additions and 375 deletions
|
|
@ -22,6 +22,7 @@ Legacy note:
|
|||
|
||||
- 2026-06-06: `maker-response-latency-and-quote-competitiveness` closed with status `passed`. Durable maker timing, relay failure classification, pair-native competitiveness surfaces, dashboard cleanup, and emergency quote-history retention fixes were implemented, validated, and deployed; long-term storage shape is opened as a separate turn.
|
||||
- 2026-06-12: `quote-lifecycle-retention-and-analytics-rollups` closed with status `paused`. Paused after retention, dashboard, semantic, and notification fixes were deployed; the next urgent proof is removing verifier salt refresh from the quote execution hot path.
|
||||
- 2026-06-12: `verifier-salt-hot-path-removal-and-executor-queue-guardrails` closed with status `passed`. Verifier current_salt RPC was removed from the quote-response hot path, cached salt failures now reject before signing or relay, queue delay and salt health are visible and alertable, and live post-deploy evidence showed cache or unavailable salt results with no new-runtime refresh rows.
|
||||
## Research Turns
|
||||
|
||||
## Planning Events
|
||||
|
|
|
|||
|
|
@ -1,235 +1,5 @@
|
|||
# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails
|
||||
# Implementation Turn
|
||||
|
||||
Status: open
|
||||
Opened: 2026-06-12
|
||||
Status: idle
|
||||
|
||||
## Goal
|
||||
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.
|
||||
|
||||
## Selected backlog items
|
||||
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.
|
||||
|
||||
## Design rules
|
||||
- This turn is execution latency and visibility only.
|
||||
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
|
||||
- Do not skip quotes because relay errors are likely.
|
||||
- Do not submit a quote response if signing prerequisites are unavailable.
|
||||
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
||||
- Keep old executor result rows readable.
|
||||
- Deploy only through repo workflow.
|
||||
|
||||
## Problem statement
|
||||
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing.
|
||||
|
||||
Live evidence showed this can block for seconds:
|
||||
|
||||
- `ae824f64-ddfd-4310-a44f-69c401fc8ad7`
|
||||
- `current_salt_source=refresh`
|
||||
- `current_salt_ms=7481.458`
|
||||
- executor received the command quickly, then blocked inside salt refresh
|
||||
- following commands waited in the single executor consumer queue:
|
||||
- `command_to_executor_ms` around `7.5s`
|
||||
- some later rows around `10-12s`
|
||||
- most failed with `quote_not_found_or_finished`
|
||||
|
||||
External verifier latency must not sit on the per-quote execution critical path.
|
||||
|
||||
## Backend changes
|
||||
|
||||
### 1. Inspect current salt and executor paths
|
||||
- Inspect:
|
||||
- `src/core/verifier-salt-cache.mjs`
|
||||
- `src/apps/trade-executor.mjs`
|
||||
- executor command expiry helpers
|
||||
- executor timing payload construction
|
||||
- runtime health and service snapshot summaries
|
||||
- ops sentinel alert generation
|
||||
- operator dashboard service/system views
|
||||
- existing verifier salt cache and executor tests
|
||||
- Identify every call path that can refresh verifier salt.
|
||||
- Confirm which call paths are quote-response hot path versus background or operator-triggered work.
|
||||
|
||||
### 2. Define salt cache contract
|
||||
- Add or refine a cache API with separate responsibilities:
|
||||
- background refresh may perform network I/O
|
||||
- hot-path read must never perform network I/O
|
||||
- Candidate API:
|
||||
- `start()` starts periodic refresh and immediate refresh attempt
|
||||
- `refreshNow()` performs explicit refresh with in-flight de-duplication
|
||||
- `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O
|
||||
- `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error
|
||||
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
|
||||
- Validate malformed salt before it can enter the cache.
|
||||
|
||||
### 3. Background refresh behavior
|
||||
- Ensure startup kicks off refresh without making the executor command handler wait on it.
|
||||
- Refresh periodically before the salt becomes stale.
|
||||
- If refresh fails:
|
||||
- keep a still-fresh cached salt usable until its age limit expires
|
||||
- expose last error and stale/unavailable state
|
||||
- do not block command handling
|
||||
- Record:
|
||||
- `last_refresh_started_at`
|
||||
- `last_refresh_completed_at`
|
||||
- `last_refresh_duration_ms`
|
||||
- `last_refresh_error`
|
||||
- `refresh_in_flight`
|
||||
- `salt_age_ms`
|
||||
- `salt_fresh`
|
||||
|
||||
### 4. Trade-executor hot path
|
||||
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read.
|
||||
- If no fresh cached salt is available:
|
||||
- publish a durable trade execution result
|
||||
- use status `rejected`
|
||||
- use explicit result code such as `verifier_salt_unavailable`
|
||||
- include salt cache state summary in the result payload
|
||||
- do not call `buildQuoteResponseSubmission`
|
||||
- do not call the relay
|
||||
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`.
|
||||
- Keep idempotency and executor state-store semantics intact.
|
||||
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.
|
||||
|
||||
### 5. Queue delay guardrails
|
||||
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`.
|
||||
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
|
||||
- Add thresholded runtime health fields:
|
||||
- recent max command-to-executor
|
||||
- p90 or bucketed queue delay if practical
|
||||
- latest command queue delay warning reason
|
||||
- Do not use queue delay to change strategy selection or pricing in this turn.
|
||||
|
||||
### 6. Runtime health and alerts
|
||||
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status.
|
||||
- Add ops-sentinel warning alerts for:
|
||||
- verifier salt stale or unavailable
|
||||
- verifier salt refresh failing
|
||||
- executor command queue delay above a conservative threshold
|
||||
- Keep alert scopes executor/service-level unless pair-specific evidence is available.
|
||||
- Do not disarm automatically unless an existing safety invariant already does so.
|
||||
|
||||
### 7. Operator dashboard
|
||||
- Surface salt cache state in the system/service view:
|
||||
- fresh/stale/unavailable
|
||||
- salt age
|
||||
- last refresh time
|
||||
- last refresh duration
|
||||
- last refresh error
|
||||
- Surface executor queue delay alongside existing trade-executor health.
|
||||
- Preserve existing lifecycle table timing fields and old-row compatibility.
|
||||
- Avoid presenting salt failures as trade PnL or settled outcomes.
|
||||
|
||||
### 8. Tests
|
||||
- Add verifier salt cache tests:
|
||||
- cached fresh read returns immediately and does not call the loader
|
||||
- missing/stale salt returns unavailable without loader call
|
||||
- background refresh records success duration and freshness
|
||||
- background refresh records error without destroying still-fresh salt
|
||||
- malformed salts fail closed
|
||||
- Add trade-executor tests or refactorable helper tests:
|
||||
- slow salt loader cannot block quote command handling
|
||||
- no cached fresh salt publishes `verifier_salt_unavailable`
|
||||
- no relay call happens when salt is unavailable
|
||||
- stale command expiry still happens before salt lookup
|
||||
- successful command uses cached salt and records `current_salt_source=cache`
|
||||
- Add runtime health and alert tests for salt and queue warnings.
|
||||
- Add dashboard/static tests for salt and queue visibility.
|
||||
|
||||
### 9. Validation and deploy
|
||||
- Run targeted tests:
|
||||
- verifier salt cache
|
||||
- trade executor
|
||||
- runtime health / ops sentinel
|
||||
- dashboard static/operator tests
|
||||
- Run full `npm test`.
|
||||
- Build operator dashboard bundle.
|
||||
- Commit with required workflow metadata.
|
||||
- Push to `main` so repo workflow deploys.
|
||||
- Do not manually patch or roll Kubernetes deployments.
|
||||
|
||||
### 10. Live evidence after deploy
|
||||
- Confirm all repo-owned deployments run the new image.
|
||||
- Query recent trade execution results:
|
||||
- `current_salt_source`
|
||||
- `current_salt_ms`
|
||||
- `command_to_executor_ms`
|
||||
- `quote_age_at_executor_receipt_ms`
|
||||
- `result_code`
|
||||
- `failure_category`
|
||||
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
|
||||
- Report command-to-executor p50/p90/p99 after rollout.
|
||||
- Report salt cache health from trade-executor state or health.
|
||||
- Confirm recent quote flow still writes decisions, commands, and results.
|
||||
|
||||
## Data and persistence
|
||||
- Use existing executor result payloads for per-command salt and queue timing.
|
||||
- Add fields additively so old rows remain readable.
|
||||
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
|
||||
- Do not add new long-term storage unless required for dashboard or alert correctness.
|
||||
|
||||
## Edge cases
|
||||
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
|
||||
- Salt cache stale: commands reject quickly, no relay call.
|
||||
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
|
||||
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
|
||||
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
|
||||
- Malformed salt from verifier: fail closed and expose error.
|
||||
- Existing stale command: reject as stale command before salt lookup.
|
||||
- Executor paused or disarmed: existing behavior remains.
|
||||
- Old result rows without salt fields remain visible.
|
||||
|
||||
## Concrete implementation order
|
||||
|
||||
### Phase 1. Audit and contracts
|
||||
- Inspect current salt cache and executor usage.
|
||||
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
|
||||
- Decide exact result code and payload shape for salt-unavailable rejection.
|
||||
|
||||
### Phase 2. Salt cache refactor
|
||||
- Implement non-blocking cached salt read.
|
||||
- Implement refresh state fields and in-flight de-duplication.
|
||||
- Ensure background refresh starts and continues.
|
||||
- Keep existing cache tests passing with updated contract.
|
||||
|
||||
### Phase 3. Executor integration
|
||||
- Change quote-response hot path to use cached salt only.
|
||||
- Add salt-unavailable rejection result.
|
||||
- Add timing/state fields for cache read and unavailable rejection.
|
||||
- Add regression tests for slow salt loader and no relay call.
|
||||
|
||||
### Phase 4. Health, alerts, dashboard
|
||||
- Expose salt freshness and queue delay in trade-executor state/health.
|
||||
- Add ops-sentinel/runtime alert tests.
|
||||
- Add dashboard visibility tests and UI fields.
|
||||
|
||||
### Phase 5. Validation and deployment
|
||||
- Run targeted tests.
|
||||
- Run full `npm test`.
|
||||
- Build dashboard bundle.
|
||||
- Commit and push.
|
||||
- Collect live post-deploy evidence.
|
||||
|
||||
## Test plan
|
||||
- `test/verifier-salt-cache.test.mjs`
|
||||
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
|
||||
- `test/runtime-health.test.mjs`
|
||||
- `test/ops-sentinel*.test.mjs`
|
||||
- `test/operator-dashboard*.test.mjs`
|
||||
- Static config tests proving no new active env knobs are added
|
||||
- Full `npm test`
|
||||
- Dashboard build
|
||||
|
||||
## Validation checklist against the proof
|
||||
- Quote-response hot path contains no verifier salt network refresh.
|
||||
- Slow `current_salt` loader does not cause multi-second command queue delay.
|
||||
- Missing/stale salt creates durable explicit rejection without relay call.
|
||||
- Salt freshness and refresh errors are operator-visible.
|
||||
- Executor queue delay is operator-visible and alertable.
|
||||
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
|
||||
- Repo workflow deploys the result.
|
||||
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.
|
||||
|
||||
## Known fakes allowed at start of this turn
|
||||
- Venue-native terminal fill ids remain unavailable.
|
||||
- Fee-complete realized PnL remains unavailable.
|
||||
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.
|
||||
No approved implementation turn is active yet.
|
||||
|
|
|
|||
145
PROOF.md
145
PROOF.md
|
|
@ -1,144 +1,5 @@
|
|||
# Implementation Proof: Verifier salt hot-path removal and executor queue guardrails
|
||||
# Implementation Proof
|
||||
|
||||
Status: open
|
||||
Opened: 2026-06-12
|
||||
Status: idle
|
||||
|
||||
## Target outcome
|
||||
Quote-response execution must not block on the NEAR verifier `current_salt` RPC.
|
||||
|
||||
The executor should use an already-fresh cached signing salt in the hot path, reject quote-response commands quickly and explicitly when no fresh salt is available, and expose enough runtime evidence for the operator to see salt freshness and command queue delay before they turn fresh quotes into stale relay failures.
|
||||
|
||||
## Why this is the next architecture test
|
||||
Live maker timing exposed a real hot-path false path:
|
||||
|
||||
- normal command-to-executor latency is usually milliseconds
|
||||
- at `2026-06-12 14:36 UTC`, one command spent about `7481 ms` refreshing `current_salt`
|
||||
- the single executor consumer then queued later commands for `7-12s`
|
||||
- those commands reached the relay after the quote was already gone and failed with `quote_not_found_or_finished`
|
||||
|
||||
The trading system cannot treat an external verifier RPC as part of per-quote execution. Execution must outpace observability, but it also must fail closed when signing prerequisites are unavailable.
|
||||
|
||||
## Scope
|
||||
- Inspect the current verifier salt cache, trade-executor quote-response hot path, command expiry guard, executor timing payloads, runtime health, ops sentinel alerts, and dashboard service views.
|
||||
- Add a non-blocking cached-salt API for quote-response execution.
|
||||
- Move verifier salt refresh responsibility fully to background refresh and explicit refresh state.
|
||||
- Ensure the quote-response hot path never performs `current_salt` RPC or waits on a salt refresh.
|
||||
- Reject commands quickly with an explicit durable result when no fresh cached salt is available.
|
||||
- Preserve existing command expiry behavior so stale commands fail closed before relay submission.
|
||||
- Surface verifier salt freshness, refresh duration/error, and executor queue delay in runtime health and dashboard/operator surfaces.
|
||||
- Add runtime alert coverage for stale or unavailable verifier salt and excessive executor queue delay.
|
||||
- Deploy only through the repo workflow and collect live evidence after rollout.
|
||||
|
||||
## Assumptions
|
||||
- A verifier salt can be reused safely within the existing freshness policy enforced by `verifier-salt-cache`.
|
||||
- The current salt cache background refresher can be extended rather than replaced.
|
||||
- A command that cannot be signed because no fresh salt is available should be rejected before relay submission; this is a signing prerequisite failure, not quote skipping to avoid relay errors.
|
||||
- The executor remains a single quote-response consumer unless implementation evidence shows concurrency is required after salt is removed from the hot path.
|
||||
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
|
||||
|
||||
## Turn-shaping rules
|
||||
- Do not change strategy selection, edge thresholds, notional limits, inventory checks, arming semantics, signer identity, pair enablement, or quote response pricing.
|
||||
- Do not skip quotes because they may produce relay errors.
|
||||
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
||||
- Do not hide or relabel relay failures as successful outcomes.
|
||||
- Do not perform manual rollout, image patching, or cluster repair; deployment must be push-driven through the repo workflow.
|
||||
- If a salt-related bug is fixed, add regression tests.
|
||||
- Preserve old executor result rows as readable even if they lack new salt fields.
|
||||
|
||||
## Non-goals
|
||||
- No new trading strategy.
|
||||
- No executor concurrency change unless required to prove the salt fix.
|
||||
- No change to quote response economics.
|
||||
- No new venue or signer.
|
||||
- No automatic disarming based on salt health unless an existing safety path already does so.
|
||||
- No realized PnL or fee attribution.
|
||||
|
||||
## Required operator behavior
|
||||
|
||||
### Salt Truth
|
||||
The operator must be able to see:
|
||||
|
||||
- whether the verifier salt cache is fresh
|
||||
- current salt age
|
||||
- last refresh start and completion time
|
||||
- last refresh duration
|
||||
- whether a refresh is in flight
|
||||
- last refresh error, if any
|
||||
- count or timestamp of commands rejected because no fresh salt was available
|
||||
|
||||
### Queue Truth
|
||||
The operator must be able to see:
|
||||
|
||||
- recent command-to-executor latency
|
||||
- whether queue delay is warning or critical
|
||||
- recent executor result timing including salt source and salt lookup duration
|
||||
- whether quote failures cluster after queue delay rather than relay latency
|
||||
|
||||
### Execution Truth
|
||||
For each quote-response result:
|
||||
|
||||
- cached salt use should be explicit in `executor_timing.current_salt_source`
|
||||
- `current_salt_source=refresh` must not appear from the quote-response hot path after this turn
|
||||
- no relay call should be attempted when a fresh cached salt is unavailable
|
||||
- missing or stale salt must produce a durable rejected result with an explicit result code
|
||||
|
||||
## Semantic invariants
|
||||
- Quote-response signing prerequisites are fail-closed.
|
||||
- No fresh cached salt means no relay submission.
|
||||
- Hot-path salt lookup is local and bounded; it never performs network I/O.
|
||||
- Background salt refresh failures are visible but do not block already-fresh cached salt usage.
|
||||
- Stale commands are still rejected before relay submission.
|
||||
- Queue delay and relay latency remain separate timing concepts.
|
||||
- Old executor result rows remain readable.
|
||||
|
||||
## Definition of done
|
||||
- `verifier-salt-cache` exposes a non-blocking fresh-salt read path for hot execution.
|
||||
- Background salt refresh maintains cache state and exposes freshness, duration, in-flight state, and errors.
|
||||
- Trade-executor quote-response handling uses only the non-blocking cached-salt path.
|
||||
- Trade-executor publishes durable rejected results when no fresh cached salt is available.
|
||||
- Executor timing payloads distinguish cached salt, unavailable salt, and historical refresh rows.
|
||||
- Runtime health and dashboard expose salt freshness and executor queue delay.
|
||||
- Ops sentinel or runtime alert tests cover stale/unavailable salt and excessive queue delay.
|
||||
- Regression tests prove slow verifier salt RPC cannot block quote-response command handling.
|
||||
- Full `npm test` passes.
|
||||
- Dashboard bundle build passes.
|
||||
- The result is deployed through repo workflow.
|
||||
- Live post-deploy evidence shows quote flow continues and command-to-executor latency is not dominated by salt refresh.
|
||||
|
||||
## Validation evidence required
|
||||
- Unit tests for non-blocking cached-salt reads.
|
||||
- Unit tests proving background refresh records duration/error and does not require hot-path waiting.
|
||||
- Trade-executor tests proving a slow `current_salt` RPC does not delay command rejection or handling.
|
||||
- Tests proving no relay submission is attempted when cached salt is missing/stale.
|
||||
- Tests proving stale command expiry still wins before signing.
|
||||
- Runtime health or ops sentinel tests for stale salt and queue-delay warnings.
|
||||
- Dashboard/static tests for salt freshness and queue-delay visibility.
|
||||
- Full `npm test`.
|
||||
- Dashboard bundle build.
|
||||
- Live evidence after deployment:
|
||||
- trade-executor deployment on the new image
|
||||
- recent executor results with `current_salt_source=cache` or explicit salt-unavailable rejection, not hot-path `refresh`
|
||||
- command-to-executor p50/p90/p99 after rollout
|
||||
- salt cache freshness state
|
||||
- recent quote flow still writing decisions, commands, and results
|
||||
|
||||
## Failure conditions
|
||||
- Any quote-response command performs verifier `current_salt` RPC in the hot path.
|
||||
- A slow salt refresh queues later commands for seconds.
|
||||
- Missing or stale salt leads to relay submission instead of fail-closed rejection.
|
||||
- Dashboard or health hides stale salt or queue delay.
|
||||
- The implementation changes strategy behavior, pricing, inventory policy, or pair enablement.
|
||||
- Deployment requires manual cluster reconciliation.
|
||||
|
||||
## Current real before this turn
|
||||
- Trade-executor has a verifier salt cache and records executor timing.
|
||||
- Live evidence showed normal command-to-executor p50 around `11 ms`, but p99 near `9954 ms` during a salt refresh stall.
|
||||
- The slow rows showed `current_salt_source=refresh` with `current_salt_ms` around `7481 ms` and `5144 ms`.
|
||||
- Subsequent queued rows reached the executor after `7-12s` and often failed with `quote_not_found_or_finished`.
|
||||
- Command expiry and maker timing fields already exist, but salt freshness and queue-delay guardrails are not operator-visible enough.
|
||||
|
||||
## Deliberately not built by this proof
|
||||
- Executor worker pool or broad concurrency model unless the non-blocking salt path is insufficient.
|
||||
- Fee-complete realized PnL.
|
||||
- Venue-native terminal fill ids.
|
||||
- Automatic strategy changes based on salt or queue metrics.
|
||||
No approved implementation proof is active yet.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,235 @@
|
|||
# Implementation Turn: Verifier salt hot-path removal and executor queue guardrails
|
||||
|
||||
Status: open
|
||||
Opened: 2026-06-12
|
||||
|
||||
## Goal
|
||||
Remove verifier `current_salt` RPC latency from quote-response execution, fail closed quickly when no fresh signing salt is available, and make salt freshness plus executor queue delay visible to operators.
|
||||
|
||||
## Selected backlog items
|
||||
- none selected; this turn was opened directly from the live latency defect observed on 2026-06-12.
|
||||
|
||||
## Design rules
|
||||
- This turn is execution latency and visibility only.
|
||||
- Do not change strategy selection, pricing, edge thresholds, notional limits, inventory checks, arming, signer identity, or pair enablement.
|
||||
- Do not skip quotes because relay errors are likely.
|
||||
- Do not submit a quote response if signing prerequisites are unavailable.
|
||||
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
||||
- Keep old executor result rows readable.
|
||||
- Deploy only through repo workflow.
|
||||
|
||||
## Problem statement
|
||||
The quote-response executor currently calls `verifierSaltCache.getFreshSalt()` from inside `handleCommand`. If the cache decides it needs to refresh, the command waits for the NEAR verifier `current_salt` RPC before signing.
|
||||
|
||||
Live evidence showed this can block for seconds:
|
||||
|
||||
- `ae824f64-ddfd-4310-a44f-69c401fc8ad7`
|
||||
- `current_salt_source=refresh`
|
||||
- `current_salt_ms=7481.458`
|
||||
- executor received the command quickly, then blocked inside salt refresh
|
||||
- following commands waited in the single executor consumer queue:
|
||||
- `command_to_executor_ms` around `7.5s`
|
||||
- some later rows around `10-12s`
|
||||
- most failed with `quote_not_found_or_finished`
|
||||
|
||||
External verifier latency must not sit on the per-quote execution critical path.
|
||||
|
||||
## Backend changes
|
||||
|
||||
### 1. Inspect current salt and executor paths
|
||||
- Inspect:
|
||||
- `src/core/verifier-salt-cache.mjs`
|
||||
- `src/apps/trade-executor.mjs`
|
||||
- executor command expiry helpers
|
||||
- executor timing payload construction
|
||||
- runtime health and service snapshot summaries
|
||||
- ops sentinel alert generation
|
||||
- operator dashboard service/system views
|
||||
- existing verifier salt cache and executor tests
|
||||
- Identify every call path that can refresh verifier salt.
|
||||
- Confirm which call paths are quote-response hot path versus background or operator-triggered work.
|
||||
|
||||
### 2. Define salt cache contract
|
||||
- Add or refine a cache API with separate responsibilities:
|
||||
- background refresh may perform network I/O
|
||||
- hot-path read must never perform network I/O
|
||||
- Candidate API:
|
||||
- `start()` starts periodic refresh and immediate refresh attempt
|
||||
- `refreshNow()` performs explicit refresh with in-flight de-duplication
|
||||
- `getCachedFreshSalt({ now, maxAgeMs })` returns fresh salt or a typed unavailable result without I/O
|
||||
- `getState()` returns freshness, age, last refresh times, duration, in-flight state, and error
|
||||
- Preserve existing public behavior where needed by non-hot-path tests, but quote-response execution must use the non-blocking API.
|
||||
- Validate malformed salt before it can enter the cache.
|
||||
|
||||
### 3. Background refresh behavior
|
||||
- Ensure startup kicks off refresh without making the executor command handler wait on it.
|
||||
- Refresh periodically before the salt becomes stale.
|
||||
- If refresh fails:
|
||||
- keep a still-fresh cached salt usable until its age limit expires
|
||||
- expose last error and stale/unavailable state
|
||||
- do not block command handling
|
||||
- Record:
|
||||
- `last_refresh_started_at`
|
||||
- `last_refresh_completed_at`
|
||||
- `last_refresh_duration_ms`
|
||||
- `last_refresh_error`
|
||||
- `refresh_in_flight`
|
||||
- `salt_age_ms`
|
||||
- `salt_fresh`
|
||||
|
||||
### 4. Trade-executor hot path
|
||||
- Replace hot-path `await verifierSaltCache.getFreshSalt()` with non-blocking cached salt read.
|
||||
- If no fresh cached salt is available:
|
||||
- publish a durable trade execution result
|
||||
- use status `rejected`
|
||||
- use explicit result code such as `verifier_salt_unavailable`
|
||||
- include salt cache state summary in the result payload
|
||||
- do not call `buildQuoteResponseSubmission`
|
||||
- do not call the relay
|
||||
- Keep command expiry before signing so stale queued commands remain rejected as `stale_execute_command`.
|
||||
- Keep idempotency and executor state-store semantics intact.
|
||||
- Remove misleading timing for salt refresh from hot-path results; after this turn hot-path timing should show local cache read or unavailable.
|
||||
|
||||
### 5. Queue delay guardrails
|
||||
- Keep recording `command_to_executor_ms` and `quote_age_at_executor_receipt_ms`.
|
||||
- Add summary state for recent command queue delay, using existing durable result payloads or in-memory rolling metrics.
|
||||
- Add thresholded runtime health fields:
|
||||
- recent max command-to-executor
|
||||
- p90 or bucketed queue delay if practical
|
||||
- latest command queue delay warning reason
|
||||
- Do not use queue delay to change strategy selection or pricing in this turn.
|
||||
|
||||
### 6. Runtime health and alerts
|
||||
- Extend trade-executor `/state` and `/health` output with salt cache freshness and queue-delay status.
|
||||
- Add ops-sentinel warning alerts for:
|
||||
- verifier salt stale or unavailable
|
||||
- verifier salt refresh failing
|
||||
- executor command queue delay above a conservative threshold
|
||||
- Keep alert scopes executor/service-level unless pair-specific evidence is available.
|
||||
- Do not disarm automatically unless an existing safety invariant already does so.
|
||||
|
||||
### 7. Operator dashboard
|
||||
- Surface salt cache state in the system/service view:
|
||||
- fresh/stale/unavailable
|
||||
- salt age
|
||||
- last refresh time
|
||||
- last refresh duration
|
||||
- last refresh error
|
||||
- Surface executor queue delay alongside existing trade-executor health.
|
||||
- Preserve existing lifecycle table timing fields and old-row compatibility.
|
||||
- Avoid presenting salt failures as trade PnL or settled outcomes.
|
||||
|
||||
### 8. Tests
|
||||
- Add verifier salt cache tests:
|
||||
- cached fresh read returns immediately and does not call the loader
|
||||
- missing/stale salt returns unavailable without loader call
|
||||
- background refresh records success duration and freshness
|
||||
- background refresh records error without destroying still-fresh salt
|
||||
- malformed salts fail closed
|
||||
- Add trade-executor tests or refactorable helper tests:
|
||||
- slow salt loader cannot block quote command handling
|
||||
- no cached fresh salt publishes `verifier_salt_unavailable`
|
||||
- no relay call happens when salt is unavailable
|
||||
- stale command expiry still happens before salt lookup
|
||||
- successful command uses cached salt and records `current_salt_source=cache`
|
||||
- Add runtime health and alert tests for salt and queue warnings.
|
||||
- Add dashboard/static tests for salt and queue visibility.
|
||||
|
||||
### 9. Validation and deploy
|
||||
- Run targeted tests:
|
||||
- verifier salt cache
|
||||
- trade executor
|
||||
- runtime health / ops sentinel
|
||||
- dashboard static/operator tests
|
||||
- Run full `npm test`.
|
||||
- Build operator dashboard bundle.
|
||||
- Commit with required workflow metadata.
|
||||
- Push to `main` so repo workflow deploys.
|
||||
- Do not manually patch or roll Kubernetes deployments.
|
||||
|
||||
### 10. Live evidence after deploy
|
||||
- Confirm all repo-owned deployments run the new image.
|
||||
- Query recent trade execution results:
|
||||
- `current_salt_source`
|
||||
- `current_salt_ms`
|
||||
- `command_to_executor_ms`
|
||||
- `quote_age_at_executor_receipt_ms`
|
||||
- `result_code`
|
||||
- `failure_category`
|
||||
- Prove no post-deploy quote-response result used `current_salt_source=refresh`.
|
||||
- Report command-to-executor p50/p90/p99 after rollout.
|
||||
- Report salt cache health from trade-executor state or health.
|
||||
- Confirm recent quote flow still writes decisions, commands, and results.
|
||||
|
||||
## Data and persistence
|
||||
- Use existing executor result payloads for per-command salt and queue timing.
|
||||
- Add fields additively so old rows remain readable.
|
||||
- Prefer in-memory runtime state for recent queue summaries unless durable aggregation is already available.
|
||||
- Do not add new long-term storage unless required for dashboard or alert correctness.
|
||||
|
||||
## Edge cases
|
||||
- Salt cache empty at startup: commands reject quickly until background refresh succeeds.
|
||||
- Salt cache stale: commands reject quickly, no relay call.
|
||||
- Salt refresh in flight: commands use still-fresh cached salt or reject; they do not wait.
|
||||
- Salt refresh fails while cached salt is still fresh: commands may continue using cached salt.
|
||||
- Salt refresh fails and cache is stale: commands reject with explicit salt-unavailable result.
|
||||
- Malformed salt from verifier: fail closed and expose error.
|
||||
- Existing stale command: reject as stale command before salt lookup.
|
||||
- Executor paused or disarmed: existing behavior remains.
|
||||
- Old result rows without salt fields remain visible.
|
||||
|
||||
## Concrete implementation order
|
||||
|
||||
### Phase 1. Audit and contracts
|
||||
- Inspect current salt cache and executor usage.
|
||||
- Add salt-cache contract tests for non-blocking cached reads and refresh state.
|
||||
- Decide exact result code and payload shape for salt-unavailable rejection.
|
||||
|
||||
### Phase 2. Salt cache refactor
|
||||
- Implement non-blocking cached salt read.
|
||||
- Implement refresh state fields and in-flight de-duplication.
|
||||
- Ensure background refresh starts and continues.
|
||||
- Keep existing cache tests passing with updated contract.
|
||||
|
||||
### Phase 3. Executor integration
|
||||
- Change quote-response hot path to use cached salt only.
|
||||
- Add salt-unavailable rejection result.
|
||||
- Add timing/state fields for cache read and unavailable rejection.
|
||||
- Add regression tests for slow salt loader and no relay call.
|
||||
|
||||
### Phase 4. Health, alerts, dashboard
|
||||
- Expose salt freshness and queue delay in trade-executor state/health.
|
||||
- Add ops-sentinel/runtime alert tests.
|
||||
- Add dashboard visibility tests and UI fields.
|
||||
|
||||
### Phase 5. Validation and deployment
|
||||
- Run targeted tests.
|
||||
- Run full `npm test`.
|
||||
- Build dashboard bundle.
|
||||
- Commit and push.
|
||||
- Collect live post-deploy evidence.
|
||||
|
||||
## Test plan
|
||||
- `test/verifier-salt-cache.test.mjs`
|
||||
- `test/trade-executor*.test.mjs` or new extracted executor helper tests
|
||||
- `test/runtime-health.test.mjs`
|
||||
- `test/ops-sentinel*.test.mjs`
|
||||
- `test/operator-dashboard*.test.mjs`
|
||||
- Static config tests proving no new active env knobs are added
|
||||
- Full `npm test`
|
||||
- Dashboard build
|
||||
|
||||
## Validation checklist against the proof
|
||||
- Quote-response hot path contains no verifier salt network refresh.
|
||||
- Slow `current_salt` loader does not cause multi-second command queue delay.
|
||||
- Missing/stale salt creates durable explicit rejection without relay call.
|
||||
- Salt freshness and refresh errors are operator-visible.
|
||||
- Executor queue delay is operator-visible and alertable.
|
||||
- Strategy behavior, edge, notional, inventory, arming, signer, and pair enablement are unchanged.
|
||||
- Repo workflow deploys the result.
|
||||
- Live results show `current_salt_source=cache` or salt-unavailable rejection, not hot-path `refresh`.
|
||||
|
||||
## Known fakes allowed at start of this turn
|
||||
- Venue-native terminal fill ids remain unavailable.
|
||||
- Fee-complete realized PnL remains unavailable.
|
||||
- Executor concurrency remains single-consumer unless proven necessary after salt hot-path removal.
|
||||
|
|
@ -0,0 +1,144 @@
|
|||
# Implementation Proof: Verifier salt hot-path removal and executor queue guardrails
|
||||
|
||||
Status: open
|
||||
Opened: 2026-06-12
|
||||
|
||||
## Target outcome
|
||||
Quote-response execution must not block on the NEAR verifier `current_salt` RPC.
|
||||
|
||||
The executor should use an already-fresh cached signing salt in the hot path, reject quote-response commands quickly and explicitly when no fresh salt is available, and expose enough runtime evidence for the operator to see salt freshness and command queue delay before they turn fresh quotes into stale relay failures.
|
||||
|
||||
## Why this is the next architecture test
|
||||
Live maker timing exposed a real hot-path false path:
|
||||
|
||||
- normal command-to-executor latency is usually milliseconds
|
||||
- at `2026-06-12 14:36 UTC`, one command spent about `7481 ms` refreshing `current_salt`
|
||||
- the single executor consumer then queued later commands for `7-12s`
|
||||
- those commands reached the relay after the quote was already gone and failed with `quote_not_found_or_finished`
|
||||
|
||||
The trading system cannot treat an external verifier RPC as part of per-quote execution. Execution must outpace observability, but it also must fail closed when signing prerequisites are unavailable.
|
||||
|
||||
## Scope
|
||||
- Inspect the current verifier salt cache, trade-executor quote-response hot path, command expiry guard, executor timing payloads, runtime health, ops sentinel alerts, and dashboard service views.
|
||||
- Add a non-blocking cached-salt API for quote-response execution.
|
||||
- Move verifier salt refresh responsibility fully to background refresh and explicit refresh state.
|
||||
- Ensure the quote-response hot path never performs `current_salt` RPC or waits on a salt refresh.
|
||||
- Reject commands quickly with an explicit durable result when no fresh cached salt is available.
|
||||
- Preserve existing command expiry behavior so stale commands fail closed before relay submission.
|
||||
- Surface verifier salt freshness, refresh duration/error, and executor queue delay in runtime health and dashboard/operator surfaces.
|
||||
- Add runtime alert coverage for stale or unavailable verifier salt and excessive executor queue delay.
|
||||
- Deploy only through the repo workflow and collect live evidence after rollout.
|
||||
|
||||
## Assumptions
|
||||
- A verifier salt can be reused safely within the existing freshness policy enforced by `verifier-salt-cache`.
|
||||
- The current salt cache background refresher can be extended rather than replaced.
|
||||
- A command that cannot be signed because no fresh salt is available should be rejected before relay submission; this is a signing prerequisite failure, not quote skipping to avoid relay errors.
|
||||
- The executor remains a single quote-response consumer unless implementation evidence shows concurrency is required after salt is removed from the hot path.
|
||||
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
|
||||
|
||||
## Turn-shaping rules
|
||||
- Do not change strategy selection, edge thresholds, notional limits, inventory checks, arming semantics, signer identity, pair enablement, or quote response pricing.
|
||||
- Do not skip quotes because they may produce relay errors.
|
||||
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
||||
- Do not hide or relabel relay failures as successful outcomes.
|
||||
- Do not perform manual rollout, image patching, or cluster repair; deployment must be push-driven through the repo workflow.
|
||||
- If a salt-related bug is fixed, add regression tests.
|
||||
- Preserve old executor result rows as readable even if they lack new salt fields.
|
||||
|
||||
## Non-goals
|
||||
- No new trading strategy.
|
||||
- No executor concurrency change unless required to prove the salt fix.
|
||||
- No change to quote response economics.
|
||||
- No new venue or signer.
|
||||
- No automatic disarming based on salt health unless an existing safety path already does so.
|
||||
- No realized PnL or fee attribution.
|
||||
|
||||
## Required operator behavior
|
||||
|
||||
### Salt Truth
|
||||
The operator must be able to see:
|
||||
|
||||
- whether the verifier salt cache is fresh
|
||||
- current salt age
|
||||
- last refresh start and completion time
|
||||
- last refresh duration
|
||||
- whether a refresh is in flight
|
||||
- last refresh error, if any
|
||||
- count or timestamp of commands rejected because no fresh salt was available
|
||||
|
||||
### Queue Truth
|
||||
The operator must be able to see:
|
||||
|
||||
- recent command-to-executor latency
|
||||
- whether queue delay is warning or critical
|
||||
- recent executor result timing including salt source and salt lookup duration
|
||||
- whether quote failures cluster after queue delay rather than relay latency
|
||||
|
||||
### Execution Truth
|
||||
For each quote-response result:
|
||||
|
||||
- cached salt use should be explicit in `executor_timing.current_salt_source`
|
||||
- `current_salt_source=refresh` must not appear from the quote-response hot path after this turn
|
||||
- no relay call should be attempted when a fresh cached salt is unavailable
|
||||
- missing or stale salt must produce a durable rejected result with an explicit result code
|
||||
|
||||
## Semantic invariants
|
||||
- Quote-response signing prerequisites are fail-closed.
|
||||
- No fresh cached salt means no relay submission.
|
||||
- Hot-path salt lookup is local and bounded; it never performs network I/O.
|
||||
- Background salt refresh failures are visible but do not block already-fresh cached salt usage.
|
||||
- Stale commands are still rejected before relay submission.
|
||||
- Queue delay and relay latency remain separate timing concepts.
|
||||
- Old executor result rows remain readable.
|
||||
|
||||
## Definition of done
|
||||
- `verifier-salt-cache` exposes a non-blocking fresh-salt read path for hot execution.
|
||||
- Background salt refresh maintains cache state and exposes freshness, duration, in-flight state, and errors.
|
||||
- Trade-executor quote-response handling uses only the non-blocking cached-salt path.
|
||||
- Trade-executor publishes durable rejected results when no fresh cached salt is available.
|
||||
- Executor timing payloads distinguish cached salt, unavailable salt, and historical refresh rows.
|
||||
- Runtime health and dashboard expose salt freshness and executor queue delay.
|
||||
- Ops sentinel or runtime alert tests cover stale/unavailable salt and excessive queue delay.
|
||||
- Regression tests prove slow verifier salt RPC cannot block quote-response command handling.
|
||||
- Full `npm test` passes.
|
||||
- Dashboard bundle build passes.
|
||||
- The result is deployed through repo workflow.
|
||||
- Live post-deploy evidence shows quote flow continues and command-to-executor latency is not dominated by salt refresh.
|
||||
|
||||
## Validation evidence required
|
||||
- Unit tests for non-blocking cached-salt reads.
|
||||
- Unit tests proving background refresh records duration/error and does not require hot-path waiting.
|
||||
- Trade-executor tests proving a slow `current_salt` RPC does not delay command rejection or handling.
|
||||
- Tests proving no relay submission is attempted when cached salt is missing/stale.
|
||||
- Tests proving stale command expiry still wins before signing.
|
||||
- Runtime health or ops sentinel tests for stale salt and queue-delay warnings.
|
||||
- Dashboard/static tests for salt freshness and queue-delay visibility.
|
||||
- Full `npm test`.
|
||||
- Dashboard bundle build.
|
||||
- Live evidence after deployment:
|
||||
- trade-executor deployment on the new image
|
||||
- recent executor results with `current_salt_source=cache` or explicit salt-unavailable rejection, not hot-path `refresh`
|
||||
- command-to-executor p50/p90/p99 after rollout
|
||||
- salt cache freshness state
|
||||
- recent quote flow still writing decisions, commands, and results
|
||||
|
||||
## Failure conditions
|
||||
- Any quote-response command performs verifier `current_salt` RPC in the hot path.
|
||||
- A slow salt refresh queues later commands for seconds.
|
||||
- Missing or stale salt leads to relay submission instead of fail-closed rejection.
|
||||
- Dashboard or health hides stale salt or queue delay.
|
||||
- The implementation changes strategy behavior, pricing, inventory policy, or pair enablement.
|
||||
- Deployment requires manual cluster reconciliation.
|
||||
|
||||
## Current real before this turn
|
||||
- Trade-executor has a verifier salt cache and records executor timing.
|
||||
- Live evidence showed normal command-to-executor p50 around `11 ms`, but p99 near `9954 ms` during a salt refresh stall.
|
||||
- The slow rows showed `current_salt_source=refresh` with `current_salt_ms` around `7481 ms` and `5144 ms`.
|
||||
- Subsequent queued rows reached the executor after `7-12s` and often failed with `quote_not_found_or_finished`.
|
||||
- Command expiry and maker timing fields already exist, but salt freshness and queue-delay guardrails are not operator-visible enough.
|
||||
|
||||
## Deliberately not built by this proof
|
||||
- Executor worker pool or broad concurrency model unless the non-blocking salt path is insufficient.
|
||||
- Fee-complete realized PnL.
|
||||
- Venue-native terminal fill ids.
|
||||
- Automatic strategy changes based on salt or queue metrics.
|
||||
Loading…
Add table
Reference in a new issue