unrip/archive/implementation/20260606T081204Z-maker-response-latency-and-quote-competitiveness-implementation.md

# Implementation Turn: Maker response latency and quote competitiveness

Status: open
Opened: 2026-05-18

## Goal
Measure and reduce maker quote-response latency after pair-native routing so nBTC/USDC responses are only sent when they are competitively viable and every missed response has durable timing evidence.

## Selected backlog items
- none selected; this turn was opened directly from the operator request on 2026-05-18.

## Design rules
- DB asset, pair, route, and strategy config remain canonical.
- No active pair, edge, notional, latency, or response-policy env vars.
- Existing nBTC/EURe behavior and 49 bps compatibility must survive.
- nBTC/USDC is the key regression pair.
- Missing pair config, missing strategy config, missing timing prerequisites, stale price, stale inventory, or invalid policy must fail closed or skip response explicitly.
- New response policy may reduce responses; it must not loosen edge, notional, inventory, arming, pair enablement, or signer checks.
- Relay acceptance remains submission evidence only.
- Deploy only through repo workflow.

## Problem statement
The previous turn removed BTC/EURe assumptions and then instrumented and optimized the executor hot path. Live evidence after verifier salt caching shows local salt lookup is no longer the normal bottleneck:

- cache-hit failed submissions are often around 45 ms p50 total
- cache-hit accepted responses are often around 86 ms p50 total
- salt timing is near 0 ms on cache hits
- relay errors such as `quote not found or already finished` still dominate

The system now needs durable timing and competitiveness truth across the full maker path. Without it, the operator cannot tell whether nBTC/USDC fails because incoming quotes are already old, because relay round trip is still too slow, because certain request kinds/notional sizes are unwinnable, or because accepted responses simply do not settle.

## Backend changes

### 1. End-to-end maker timing model
- Define a normalized maker timing object carried by quotes, decisions, commands, results, and lifecycle rows where available:
  - `quote_observed_at`
  - `quote_received_at`
  - `quote_normalized_at`
  - `quote_published_at`
  - `strategy_received_at`
  - `strategy_decided_at`
  - `command_published_at`
  - `executor_received_at`
  - `relay_result_at`
  - `outcome_observed_at`
  - `quote_to_decision_ms`
  - `decision_to_command_ms`
  - `command_to_executor_ms`
  - `executor_to_relay_result_ms`
  - `quote_to_relay_result_ms`
  - `quote_to_outcome_ms`
- Use in-process monotonic timers where one process owns both timestamps.
- Use wall-clock timestamps for cross-service timing and mark impossible/negative values as unavailable instead of silently reporting misleading latency.
- Preserve the existing `executor_timing` payload and extend it rather than replacing it.
- Keep old rows readable when new timing fields are absent.

### 2. Quote ingress and normalization timestamps
- Inspect the NEAR Intents ingest path and normalized swap-demand event builder.
- Add timestamps at the earliest reliable point where the raw quote frame is received.
- Carry quote receive/publish timestamps into normalized quote payloads.
- Ensure history writer persists the timing fields without requiring a schema rewrite.
- Add tests proving raw/normalized timing survives quote normalization and durable history insertion.

### 3. Strategy and command timing propagation
- Update strategy decision payloads to carry quote timing metadata and strategy receipt/decision timestamps.
- Update execute command payloads to carry:
  - quote timing metadata
  - strategy timing metadata
  - command publish timestamp
  - pair id, route id, request kind, notional symbol, and notional bucket inputs needed for aggregation
- Ensure strategy rejection decisions also persist quote age and policy context.
- Add tests for nBTC/EURe compatibility and nBTC/USDC timing propagation.

### 4. Executor timing extension
- Extend executor result payloads with:
  - executor received timestamp
  - quote age at executor receipt
  - quote age at relay result
  - salt source and salt age from the existing verifier salt cache
  - relay result timing and relay error text
- Keep executor duplicate detection and stale-command rejection distinct from strategy/policy skips.
- Add tests proving executor timing is present on submitted, failed, stale, and disarmed paths where a result is emitted.

### 5. Relay failure classification
- Normalize relay submission failures into durable categories without losing the original error:
  - `quote_not_found_or_finished`
  - `relay_timeout`
  - `relay_disconnected`
  - `signing_failed`
  - `salt_unavailable`
  - `unknown_submission_failure`
- Keep `result_code=submission_failed` compatibility where needed, but add a more specific `failure_category`.
- Add tests for known relay error strings and old-row compatibility.

### 6. Competitiveness aggregation
- Add shared aggregation helpers for recent maker response competitiveness:
  - counts by pair, direction, request kind, result code, and failure category
  - latency percentiles by stage
  - quote-age buckets at decision, executor receipt, and relay result
  - accepted response rate by age bucket
  - not-filled/completed/submitted outcome rate by age bucket where outcome data exists
  - notional buckets per notional asset
- Prefer deriving summaries from durable DB rows for dashboard bootstrap.
- Keep live WebSocket updates incremental enough that the dashboard does not require full refresh for every quote.
- Add unit tests for aggregation fixtures covering nBTC/EURe and nBTC/USDC.

### 7. DB-backed response-age policy
- After the timing fields and summaries exist, add a conservative pair-scoped maker response-age policy if evidence supports it.
- Store policy in DB-owned strategy config or an adjacent DB config table, not env vars.
- Candidate fields:
  - `maker_max_quote_age_ms`
  - `maker_max_quote_age_enabled`
  - `maker_latency_policy_reason`
  - config version and updated timestamp
- Default behavior must preserve current nBTC/EURe unless a stricter policy is explicitly configured.
- For nBTC/USDC, configure only a skip policy that reduces stale/unwinnable submissions; do not reduce edge or increase notional.
- When policy skips a quote, persist a strategy rejection/block with:
  - `decision=blocked` or existing compatible rejection vocabulary
  - `decision_reason=maker_quote_too_old`
  - measured quote age
  - configured max age
  - pair config id/version
  - route id/reference price id
- Add tests proving:
  - old quotes are skipped before command emission
  - fresh quotes continue through the existing checks
  - missing/invalid policy does not create a global responder
  - policy is pair-scoped
  - skip rows appear in lifecycle without being labeled relay failures

### 8. Dashboard and operator surfaces
- Add a maker competitiveness section to the Strategy page or System page, whichever matches existing layout best after inspection.
- Show:
  - pair/direction/request-kind filters
  - accepted vs failed relay responses
  - `quote_not_found_or_finished` counts
  - p50/p90/p99 quote-to-relay result timing
  - salt source distribution
  - age-bucket outcome table
  - latest exact relay errors with quote ids
  - policy skip counts and reasons
- Extend existing lifecycle details to show the full timing waterfall for a selected quote.
- Avoid hardcoded EURe/BTC labels; use pair-native symbols already exposed by the previous turn.
- Add dashboard static and data tests.

### 9. Runtime health and alerts
- Add a pair-scoped warning when a configured maker pair has high recent stale/already-finished response rate after enough sample volume.
- Keep alerts informational/warning by default; do not auto-pause or auto-disarm unless an existing critical safety invariant is broken.
- Ensure ops-sentinel/service health summaries include enough timing state to diagnose stale response pressure by pair.
- Add tests for pair-scoped high-failure summaries and no global active-pair regression.

### 10. Deployment and live validation
- Run targeted tests for timing propagation, failure classification, aggregation, policy skip behavior, dashboard surfaces, and config/static deployment invariants.
- Run full `npm test`.
- Build the operator dashboard bundle.
- Deploy only through repo workflow.
- After deploy, collect live evidence:
  - timing distribution before/after policy if a policy was enabled
  - nBTC/USDC response counts by result category
  - policy skip counts
  - relay accepted response rate by age bucket
  - outcome status distribution for accepted responses

## Data and persistence
- Prefer additive JSON payload fields and derived summaries over destructive schema changes.
- If a schema change is needed for policy config, make it idempotent and preserve old rows.
- Keep raw and normalized event payloads replayable.
- Store original relay error messages alongside normalized categories.
- Keep timing fields nullable for old history and unavailable timestamps.

## Edge cases
- Quote has no observed timestamp: fail closed for age-based policy or mark timing unavailable; do not invent zero age.
- Cross-service clock skew yields negative duration: expose timing as unavailable with a reason.
- Strategy rejects before executor: lifecycle shows strategy/policy skip, not relay failure.
- Executor duplicate skip: does not count as a fresh relay response.
- Salt cache stale/unavailable: signing fails closed with `salt_unavailable` or existing compatible failure plus category.
- Relay disconnected or timeout: category is distinct from `quote_not_found_or_finished`.
- Pair is observe-only or disabled: no response policy makes it tradeable.
- Multiple maker-enabled pairs: summaries and alerts remain pair-scoped.
- Old BTC/EURe rows without timing remain readable.

## Concrete implementation order

### Phase 1. Codebase inspection and timing contracts
- Inspect current NEAR Intents ingest, normalized quote, strategy decision, command, executor result, history writer, dashboard lifecycle, and ops-sentinel paths.
- Write the canonical timing-field contract and helper functions.
- Add low-level tests for duration calculation and clock-skew handling.

### Phase 2. Timing propagation
- Add quote ingress timestamps.
- Propagate timing through strategy decisions and commands.
- Extend executor timing with quote age fields.
- Add propagation tests and old-row compatibility tests.

### Phase 3. Failure classification and aggregation
- Normalize relay failure categories.
- Build durable/live competitiveness summary helpers.
- Add nBTC/EURe and nBTC/USDC aggregation tests.

### Phase 4. Operator UI
- Add timing waterfall to lifecycle details.
- Add pair-native competitiveness summary with filters and age buckets.
- Add dashboard tests and build the bundle.

### Phase 5. Policy gate
- Add DB-backed pair-scoped `maker_max_quote_age_ms` policy only after summaries exist.
- Wire policy into strategy before command emission.
- Persist visible skip reasons.
- Add fail-closed and pair-scoped policy tests.

### Phase 6. Alerts and validation
- Add pair-scoped high stale/already-finished response warning.
- Run targeted tests, full `npm test`, and dashboard build.
- Deploy through repo workflow.
- Validate live timing and response distributions after rollout.

## Test plan
- Timing helper unit tests.
- Quote normalization timing tests.
- Strategy timing propagation tests.
- Executor timing extension tests.
- Relay failure classification tests.
- Competitiveness aggregation tests for nBTC/EURe and nBTC/USDC.
- Response-age policy tests:
  - old quote skipped
  - fresh quote allowed
  - missing policy does not loosen behavior
  - invalid policy fails closed
  - pair-scoped behavior
- Dashboard lifecycle and competitiveness UI tests.
- Runtime-health/alert pair-scope tests.
- Static deployment tests proving no new pair/latency env vars.
- Full `npm test`.
- Dashboard build.

## Validation checklist against the proof
- A selected nBTC/USDC quote can be explained from quote ingress to relay result with durable timing fields.
- `quote not found or already finished` is categorized and shown with exact error text.
- Dashboard summarizes accepted/failed/skipped responses by pair, direction, request kind, and age bucket.
- Policy skips, if enabled, are visible as strategy/policy skips and not relay failures.
- nBTC/EURe behavior and 49 bps compatibility remain intact.
- No new live trading is enabled by import or env vars.
- Repo workflow deploys the result without manual cluster reconciliation.

## Known fakes allowed at start of this turn
- Fee-complete realized PnL remains unavailable.
- Venue-native terminal fill ids remain unavailable unless existing relay/status evidence exposes them.
- Arbitrary low-latency infrastructure, multi-region placement, and multi-executor racing are not built.
- Quote competitiveness is initially observational until the timing data supports a concrete skip policy.