unrip/archive/implementation/20260606T081204Z-maker-response-latency-and-quote-competitiveness-implementation.md

Implementation Turn: Maker response latency and quote competitiveness

Status: open Opened: 2026-05-18

Goal

Measure and reduce maker quote-response latency after pair-native routing so nBTC/USDC responses are only sent when they are competitively viable and every missed response has durable timing evidence.

Selected backlog items

• none selected; this turn was opened directly from the operator request on 2026-05-18.

Design rules

• DB asset, pair, route, and strategy config remain canonical.
• No active pair, edge, notional, latency, or response-policy env vars.
• Existing nBTC/EURe behavior and 49 bps compatibility must survive.
• nBTC/USDC is the key regression pair.
• Missing pair config, missing strategy config, missing timing prerequisites, stale price, stale inventory, or invalid policy must fail closed or skip response explicitly.
• New response policy may reduce responses; it must not loosen edge, notional, inventory, arming, pair enablement, or signer checks.
• Relay acceptance remains submission evidence only.
• Deploy only through repo workflow.

Problem statement

The previous turn removed BTC/EURe assumptions and then instrumented and optimized the executor hot path. Live evidence after verifier salt caching shows local salt lookup is no longer the normal bottleneck:

• cache-hit failed submissions are often around 45 ms p50 total
• cache-hit accepted responses are often around 86 ms p50 total
• salt timing is near 0 ms on cache hits
• relay errors such as quote not found or already finished still dominate

The system now needs durable timing and competitiveness truth across the full maker path. Without it, the operator cannot tell whether nBTC/USDC fails because incoming quotes are already old, because relay round trip is still too slow, because certain request kinds/notional sizes are unwinnable, or because accepted responses simply do not settle.

Backend changes

1. End-to-end maker timing model

• Define a normalized maker timing object carried by quotes, decisions, commands, results, and lifecycle rows where available:
  • quote_observed_at
  • quote_received_at
  • quote_normalized_at
  • quote_published_at
  • strategy_received_at
  • strategy_decided_at
  • command_published_at
  • executor_received_at
  • relay_result_at
  • outcome_observed_at
  • quote_to_decision_ms
  • decision_to_command_ms
  • command_to_executor_ms
  • executor_to_relay_result_ms
  • quote_to_relay_result_ms
  • quote_to_outcome_ms
• Use in-process monotonic timers where one process owns both timestamps.
• Use wall-clock timestamps for cross-service timing and mark impossible/negative values as unavailable instead of silently reporting misleading latency.
• Preserve the existing executor_timing payload and extend it rather than replacing it.
• Keep old rows readable when new timing fields are absent.

2. Quote ingress and normalization timestamps

• Inspect the NEAR Intents ingest path and normalized swap-demand event builder.
• Add timestamps at the earliest reliable point where the raw quote frame is received.
• Carry quote receive/publish timestamps into normalized quote payloads.
• Ensure history writer persists the timing fields without requiring a schema rewrite.
• Add tests proving raw/normalized timing survives quote normalization and durable history insertion.

3. Strategy and command timing propagation

• Update strategy decision payloads to carry quote timing metadata and strategy receipt/decision timestamps.
• Update execute command payloads to carry:
  • quote timing metadata
  • strategy timing metadata
  • command publish timestamp
  • pair id, route id, request kind, notional symbol, and notional bucket inputs needed for aggregation
• Ensure strategy rejection decisions also persist quote age and policy context.
• Add tests for nBTC/EURe compatibility and nBTC/USDC timing propagation.

4. Executor timing extension

• Extend executor result payloads with:
  • executor received timestamp
  • quote age at executor receipt
  • quote age at relay result
  • salt source and salt age from the existing verifier salt cache
  • relay result timing and relay error text
• Keep executor duplicate detection and stale-command rejection distinct from strategy/policy skips.
• Add tests proving executor timing is present on submitted, failed, stale, and disarmed paths where a result is emitted.

5. Relay failure classification

• Normalize relay submission failures into durable categories without losing the original error:
  • quote_not_found_or_finished
  • relay_timeout
  • relay_disconnected
  • signing_failed
  • salt_unavailable
  • unknown_submission_failure
• Keep result_code=submission_failed compatibility where needed, but add a more specific failure_category.
• Add tests for known relay error strings and old-row compatibility.

6. Competitiveness aggregation

• Add shared aggregation helpers for recent maker response competitiveness:
  • counts by pair, direction, request kind, result code, and failure category
  • latency percentiles by stage
  • quote-age buckets at decision, executor receipt, and relay result
  • accepted response rate by age bucket
  • not-filled/completed/submitted outcome rate by age bucket where outcome data exists
  • notional buckets per notional asset
• Prefer deriving summaries from durable DB rows for dashboard bootstrap.
• Keep live WebSocket updates incremental enough that the dashboard does not require full refresh for every quote.
• Add unit tests for aggregation fixtures covering nBTC/EURe and nBTC/USDC.

7. DB-backed response-age policy

• After the timing fields and summaries exist, add a conservative pair-scoped maker response-age policy if evidence supports it.
• Store policy in DB-owned strategy config or an adjacent DB config table, not env vars.
• Candidate fields:
  • maker_max_quote_age_ms
  • maker_max_quote_age_enabled
  • maker_latency_policy_reason
  • config version and updated timestamp
• Default behavior must preserve current nBTC/EURe unless a stricter policy is explicitly configured.
• For nBTC/USDC, configure only a skip policy that reduces stale/unwinnable submissions; do not reduce edge or increase notional.
• When policy skips a quote, persist a strategy rejection/block with:
  • decision=blocked or existing compatible rejection vocabulary
  • decision_reason=maker_quote_too_old
  • measured quote age
  • configured max age
  • pair config id/version
  • route id/reference price id
• Add tests proving:
  • old quotes are skipped before command emission
  • fresh quotes continue through the existing checks
  • missing/invalid policy does not create a global responder
  • policy is pair-scoped
  • skip rows appear in lifecycle without being labeled relay failures

8. Dashboard and operator surfaces

• Add a maker competitiveness section to the Strategy page or System page, whichever matches existing layout best after inspection.
• Show:
  • pair/direction/request-kind filters
  • accepted vs failed relay responses
  • quote_not_found_or_finished counts
  • p50/p90/p99 quote-to-relay result timing
  • salt source distribution
  • age-bucket outcome table
  • latest exact relay errors with quote ids
  • policy skip counts and reasons
• Extend existing lifecycle details to show the full timing waterfall for a selected quote.
• Avoid hardcoded EURe/BTC labels; use pair-native symbols already exposed by the previous turn.
• Add dashboard static and data tests.

9. Runtime health and alerts

• Add a pair-scoped warning when a configured maker pair has high recent stale/already-finished response rate after enough sample volume.
• Keep alerts informational/warning by default; do not auto-pause or auto-disarm unless an existing critical safety invariant is broken.
• Ensure ops-sentinel/service health summaries include enough timing state to diagnose stale response pressure by pair.
• Add tests for pair-scoped high-failure summaries and no global active-pair regression.

10. Deployment and live validation

• Run targeted tests for timing propagation, failure classification, aggregation, policy skip behavior, dashboard surfaces, and config/static deployment invariants.
• Run full npm test.
• Build the operator dashboard bundle.
• Deploy only through repo workflow.
• After deploy, collect live evidence:
  • timing distribution before/after policy if a policy was enabled
  • nBTC/USDC response counts by result category
  • policy skip counts
  • relay accepted response rate by age bucket
  • outcome status distribution for accepted responses

Data and persistence

• Prefer additive JSON payload fields and derived summaries over destructive schema changes.
• If a schema change is needed for policy config, make it idempotent and preserve old rows.
• Keep raw and normalized event payloads replayable.
• Store original relay error messages alongside normalized categories.
• Keep timing fields nullable for old history and unavailable timestamps.

Edge cases

• Quote has no observed timestamp: fail closed for age-based policy or mark timing unavailable; do not invent zero age.
• Cross-service clock skew yields negative duration: expose timing as unavailable with a reason.
• Strategy rejects before executor: lifecycle shows strategy/policy skip, not relay failure.
• Executor duplicate skip: does not count as a fresh relay response.
• Salt cache stale/unavailable: signing fails closed with salt_unavailable or existing compatible failure plus category.
• Relay disconnected or timeout: category is distinct from quote_not_found_or_finished.
• Pair is observe-only or disabled: no response policy makes it tradeable.
• Multiple maker-enabled pairs: summaries and alerts remain pair-scoped.
• Old BTC/EURe rows without timing remain readable.

Concrete implementation order

Phase 1. Codebase inspection and timing contracts

• Inspect current NEAR Intents ingest, normalized quote, strategy decision, command, executor result, history writer, dashboard lifecycle, and ops-sentinel paths.
• Write the canonical timing-field contract and helper functions.
• Add low-level tests for duration calculation and clock-skew handling.

Phase 2. Timing propagation

• Add quote ingress timestamps.
• Propagate timing through strategy decisions and commands.
• Extend executor timing with quote age fields.
• Add propagation tests and old-row compatibility tests.

Phase 3. Failure classification and aggregation

• Normalize relay failure categories.
• Build durable/live competitiveness summary helpers.
• Add nBTC/EURe and nBTC/USDC aggregation tests.

Phase 4. Operator UI

• Add timing waterfall to lifecycle details.
• Add pair-native competitiveness summary with filters and age buckets.
• Add dashboard tests and build the bundle.

Phase 5. Policy gate

• Add DB-backed pair-scoped maker_max_quote_age_ms policy only after summaries exist.
• Wire policy into strategy before command emission.
• Persist visible skip reasons.
• Add fail-closed and pair-scoped policy tests.

Phase 6. Alerts and validation

• Add pair-scoped high stale/already-finished response warning.
• Run targeted tests, full npm test, and dashboard build.
• Deploy through repo workflow.
• Validate live timing and response distributions after rollout.

Test plan

• Timing helper unit tests.
• Quote normalization timing tests.
• Strategy timing propagation tests.
• Executor timing extension tests.
• Relay failure classification tests.
• Competitiveness aggregation tests for nBTC/EURe and nBTC/USDC.
• Response-age policy tests:
  • old quote skipped
  • fresh quote allowed
  • missing policy does not loosen behavior
  • invalid policy fails closed
  • pair-scoped behavior
• Dashboard lifecycle and competitiveness UI tests.
• Runtime-health/alert pair-scope tests.
• Static deployment tests proving no new pair/latency env vars.
• Full npm test.
• Dashboard build.

Validation checklist against the proof

• A selected nBTC/USDC quote can be explained from quote ingress to relay result with durable timing fields.
• quote not found or already finished is categorized and shown with exact error text.
• Dashboard summarizes accepted/failed/skipped responses by pair, direction, request kind, and age bucket.
• Policy skips, if enabled, are visible as strategy/policy skips and not relay failures.
• nBTC/EURe behavior and 49 bps compatibility remain intact.
• No new live trading is enabled by import or env vars.
• Repo workflow deploys the result without manual cluster reconciliation.

Known fakes allowed at start of this turn

• Fee-complete realized PnL remains unavailable.
• Venue-native terminal fill ids remain unavailable unless existing relay/status evidence exposes them.
• Arbitrary low-latency infrastructure, multi-region placement, and multi-executor racing are not built.
• Quote competitiveness is initially observational until the timing data supports a concrete skip policy.