cf90597678
Proof: Preserve the completed implementation turn and record its outcome in the tracked archive. Assumptions: The archived files capture the relevant planning state for the completed turn. Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.
144 lines
8.4 KiB
Markdown
144 lines
8.4 KiB
Markdown
# Implementation Proof: Verifier salt hot-path removal and executor queue guardrails
|
|
|
|
Status: open
|
|
Opened: 2026-06-12
|
|
|
|
## Target outcome
|
|
Quote-response execution must not block on the NEAR verifier `current_salt` RPC.
|
|
|
|
The executor should use an already-fresh cached signing salt in the hot path, reject quote-response commands quickly and explicitly when no fresh salt is available, and expose enough runtime evidence for the operator to see salt freshness and command queue delay before they turn fresh quotes into stale relay failures.
|
|
|
|
## Why this is the next architecture test
|
|
Live maker timing exposed a real hot-path false path:
|
|
|
|
- normal command-to-executor latency is usually milliseconds
|
|
- at `2026-06-12 14:36 UTC`, one command spent about `7481 ms` refreshing `current_salt`
|
|
- the single executor consumer then queued later commands for `7-12s`
|
|
- those commands reached the relay after the quote was already gone and failed with `quote_not_found_or_finished`
|
|
|
|
The trading system cannot treat an external verifier RPC as part of per-quote execution. Execution must outpace observability, but it also must fail closed when signing prerequisites are unavailable.
|
|
|
|
## Scope
|
|
- Inspect the current verifier salt cache, trade-executor quote-response hot path, command expiry guard, executor timing payloads, runtime health, ops sentinel alerts, and dashboard service views.
|
|
- Add a non-blocking cached-salt API for quote-response execution.
|
|
- Move verifier salt refresh responsibility fully to background refresh and explicit refresh state.
|
|
- Ensure the quote-response hot path never performs `current_salt` RPC or waits on a salt refresh.
|
|
- Reject commands quickly with an explicit durable result when no fresh cached salt is available.
|
|
- Preserve existing command expiry behavior so stale commands fail closed before relay submission.
|
|
- Surface verifier salt freshness, refresh duration/error, and executor queue delay in runtime health and dashboard/operator surfaces.
|
|
- Add runtime alert coverage for stale or unavailable verifier salt and excessive executor queue delay.
|
|
- Deploy only through the repo workflow and collect live evidence after rollout.
|
|
|
|
## Assumptions
|
|
- A verifier salt can be reused safely within the existing freshness policy enforced by `verifier-salt-cache`.
|
|
- The current salt cache background refresher can be extended rather than replaced.
|
|
- A command that cannot be signed because no fresh salt is available should be rejected before relay submission; this is a signing prerequisite failure, not quote skipping to avoid relay errors.
|
|
- The executor remains a single quote-response consumer unless implementation evidence shows concurrency is required after salt is removed from the hot path.
|
|
- Venue-native terminal fill ids and fee-complete realized PnL remain unavailable.
|
|
|
|
## Turn-shaping rules
|
|
- Do not change strategy selection, edge thresholds, notional limits, inventory checks, arming semantics, signer identity, pair enablement, or quote response pricing.
|
|
- Do not skip quotes because they may produce relay errors.
|
|
- Do not add active pair, edge, notional, latency, response-policy, or retention env vars.
|
|
- Do not hide or relabel relay failures as successful outcomes.
|
|
- Do not perform manual rollout, image patching, or cluster repair; deployment must be push-driven through the repo workflow.
|
|
- If a salt-related bug is fixed, add regression tests.
|
|
- Preserve old executor result rows as readable even if they lack new salt fields.
|
|
|
|
## Non-goals
|
|
- No new trading strategy.
|
|
- No executor concurrency change unless required to prove the salt fix.
|
|
- No change to quote response economics.
|
|
- No new venue or signer.
|
|
- No automatic disarming based on salt health unless an existing safety path already does so.
|
|
- No realized PnL or fee attribution.
|
|
|
|
## Required operator behavior
|
|
|
|
### Salt Truth
|
|
The operator must be able to see:
|
|
|
|
- whether the verifier salt cache is fresh
|
|
- current salt age
|
|
- last refresh start and completion time
|
|
- last refresh duration
|
|
- whether a refresh is in flight
|
|
- last refresh error, if any
|
|
- count or timestamp of commands rejected because no fresh salt was available
|
|
|
|
### Queue Truth
|
|
The operator must be able to see:
|
|
|
|
- recent command-to-executor latency
|
|
- whether queue delay is warning or critical
|
|
- recent executor result timing including salt source and salt lookup duration
|
|
- whether quote failures cluster after queue delay rather than relay latency
|
|
|
|
### Execution Truth
|
|
For each quote-response result:
|
|
|
|
- cached salt use should be explicit in `executor_timing.current_salt_source`
|
|
- `current_salt_source=refresh` must not appear from the quote-response hot path after this turn
|
|
- no relay call should be attempted when a fresh cached salt is unavailable
|
|
- missing or stale salt must produce a durable rejected result with an explicit result code
|
|
|
|
## Semantic invariants
|
|
- Quote-response signing prerequisites are fail-closed.
|
|
- No fresh cached salt means no relay submission.
|
|
- Hot-path salt lookup is local and bounded; it never performs network I/O.
|
|
- Background salt refresh failures are visible but do not block already-fresh cached salt usage.
|
|
- Stale commands are still rejected before relay submission.
|
|
- Queue delay and relay latency remain separate timing concepts.
|
|
- Old executor result rows remain readable.
|
|
|
|
## Definition of done
|
|
- `verifier-salt-cache` exposes a non-blocking fresh-salt read path for hot execution.
|
|
- Background salt refresh maintains cache state and exposes freshness, duration, in-flight state, and errors.
|
|
- Trade-executor quote-response handling uses only the non-blocking cached-salt path.
|
|
- Trade-executor publishes durable rejected results when no fresh cached salt is available.
|
|
- Executor timing payloads distinguish cached salt, unavailable salt, and historical refresh rows.
|
|
- Runtime health and dashboard expose salt freshness and executor queue delay.
|
|
- Ops sentinel or runtime alert tests cover stale/unavailable salt and excessive queue delay.
|
|
- Regression tests prove slow verifier salt RPC cannot block quote-response command handling.
|
|
- Full `npm test` passes.
|
|
- Dashboard bundle build passes.
|
|
- The result is deployed through repo workflow.
|
|
- Live post-deploy evidence shows quote flow continues and command-to-executor latency is not dominated by salt refresh.
|
|
|
|
## Validation evidence required
|
|
- Unit tests for non-blocking cached-salt reads.
|
|
- Unit tests proving background refresh records duration/error and does not require hot-path waiting.
|
|
- Trade-executor tests proving a slow `current_salt` RPC does not delay command rejection or handling.
|
|
- Tests proving no relay submission is attempted when cached salt is missing/stale.
|
|
- Tests proving stale command expiry still wins before signing.
|
|
- Runtime health or ops sentinel tests for stale salt and queue-delay warnings.
|
|
- Dashboard/static tests for salt freshness and queue-delay visibility.
|
|
- Full `npm test`.
|
|
- Dashboard bundle build.
|
|
- Live evidence after deployment:
|
|
- trade-executor deployment on the new image
|
|
- recent executor results with `current_salt_source=cache` or explicit salt-unavailable rejection, not hot-path `refresh`
|
|
- command-to-executor p50/p90/p99 after rollout
|
|
- salt cache freshness state
|
|
- recent quote flow still writing decisions, commands, and results
|
|
|
|
## Failure conditions
|
|
- Any quote-response command performs verifier `current_salt` RPC in the hot path.
|
|
- A slow salt refresh queues later commands for seconds.
|
|
- Missing or stale salt leads to relay submission instead of fail-closed rejection.
|
|
- Dashboard or health hides stale salt or queue delay.
|
|
- The implementation changes strategy behavior, pricing, inventory policy, or pair enablement.
|
|
- Deployment requires manual cluster reconciliation.
|
|
|
|
## Current real before this turn
|
|
- Trade-executor has a verifier salt cache and records executor timing.
|
|
- Live evidence showed normal command-to-executor p50 around `11 ms`, but p99 near `9954 ms` during a salt refresh stall.
|
|
- The slow rows showed `current_salt_source=refresh` with `current_salt_ms` around `7481 ms` and `5144 ms`.
|
|
- Subsequent queued rows reached the executor after `7-12s` and often failed with `quote_not_found_or_finished`.
|
|
- Command expiry and maker timing fields already exist, but salt freshness and queue-delay guardrails are not operator-visible enough.
|
|
|
|
## Deliberately not built by this proof
|
|
- Executor worker pool or broad concurrency model unless the non-blocking salt path is insufficient.
|
|
- Fee-complete realized PnL.
|
|
- Venue-native terminal fill ids.
|
|
- Automatic strategy changes based on salt or queue metrics.
|