unrip/archive/implementation/20260606T081204Z-maker-response-latency-and-quote-competitiveness-proof.md
philipp fb547f24d9
All checks were successful
deploy / deploy (push) Successful in 48s
fix: scope inventory fetch by submission time window to prevent OOM
Instead of fetching up to 50k inventory snapshots (~442MB), scope the
query to only the time range of the current submission batch with a
15-minute buffer. For a typical 1-hour batch this drops from 50k rows
to ~300 rows, well within the 1280Mi pod memory limit.

The coalesced_at_idx on intent_inventory_snapshots covers the BETWEEN
clause so this remains efficient.

Proof: history-writer OOM kills from refreshQuoteOutcomes inventory fetch
Assumptions: 15min buffer covers the attribution window for all submissions
Still fake: heuristic gap outcomes may attribute trades imprecisely
2026-06-16 17:17:19 +02:00

9.7 KiB

Implementation Proof: Maker response latency and quote competitiveness

Status: open Opened: 2026-05-18

Target outcome

The maker path must explain why nBTC/USDC quote responses are accepted, rejected by the relay, or never become settled trades.

After the pair-native turn, the local executor hot path is fast enough that blind optimization is no longer justified. This turn succeeds when the repository records an end-to-end timing waterfall for each responded quote, exposes pair/request-kind competitiveness metrics to the operator, and uses DB-backed response policy to skip quotes that are already too old or otherwise empirically unwinnable.

Why this is the next architecture test

Live evidence after verifier salt caching showed cache-hit executor results around:

  • submission_failed: p50 total near 45 ms, p90 near 55 ms, p50 salt near 0 ms
  • quote_response_ok: p50 total near 86 ms, p90 near 141 ms, p50 salt near 0 ms
  • occasional blocking salt refreshes still exist, but most local salt latency is gone

Despite that, nBTC/USDC still produces many relay errors such as quote not found or already finished, and accepted relay responses are not reliably becoming durable settlement evidence.

The next question is not "can the code answer quotes at all"; it is whether each response is competitive at the moment we answer it, and whether the bot should skip quotes whose measured age and route characteristics make them unlikely to fill.

Scope

  • Add durable, pair-native maker timing fields from quote ingress through strategy, command, executor signing, relay response, and outcome attribution.
  • Preserve existing executor timing fields and extend them with quote-age and stage timing that can be aggregated across services.
  • Classify relay responses and failures by:
    • pair id
    • source/destination assets
    • request kind
    • notional asset and notional bucket
    • route id/reference price id
    • quote age at decision, command, executor receipt, and relay response
    • relay result code and exact relay error message
    • eventual outcome status when available
  • Add operator-visible competitiveness metrics for nBTC/EURe and nBTC/USDC:
    • response counts
    • accepted response rate
    • stale/already-finished response rate
    • p50/p90/p99 stage latency
    • quote age bucket outcomes
    • pair/direction/request-kind breakdowns
  • Add a DB-backed maker response-age policy only if evidence from the new timing data supports it.
  • Keep response-age policy pair-scoped and fail closed:
    • missing pair config must not create a default live responder
    • missing or invalid max-age config must not loosen behavior
    • age cutoffs may skip responses but must not reduce edge or raise notional limits
  • Preserve the existing nBTC/EURe behavior unless the operator explicitly configures a stricter pair policy.
  • Keep nBTC/USDC as the key regression pair.
  • Deploy only through the repo workflow.

Assumptions

  • NEAR Intents quote availability can expire or be consumed independently of our local execution speed.
  • The relay error quote not found or already finished is ambiguous; it may mean quote expiry, competing solver completion, or a relay-side state transition.
  • Wall-clock timestamps across pods are good enough for operator timing ranges, but in-process monotonic timings remain the source of truth for sub-step latency.
  • The active proof may reduce responses to empirically stale quotes, but it must not increase live-funds exposure, notional size, edge looseness, or automatic pair enablement.
  • Postgres remains the canonical runtime source for pair and strategy config.

Turn-shaping rules

  • Do not add new env vars for latency policy, pair selection, or trade sizing.
  • Do not relax edge thresholds, notional limits, inventory checks, arming, or pair enablement.
  • Do not create new live trading by import or by default policy rows.
  • Do not treat relay acceptance as a completed trade.
  • Do not claim quote competitiveness from sampled logs alone; persist the fields needed to recompute it.
  • Do not do manual deployment or cluster repair outside the repo workflow.
  • If a policy change skips responses, the dashboard must show the skip reason explicitly.

Non-goals

  • No new execution venue.
  • No geographic relocation, paid low-latency infrastructure, or permanent infra spend.
  • No multi-replica executor race strategy.
  • No response flooding or duplicate submissions.
  • No automatic edge reduction to win more quotes.
  • No fee-complete realized PnL.
  • No venue-native terminal fill integration beyond existing evidence paths.

Required operator behavior

Quote timing truth

For a quote that reaches strategy and executor, the operator must be able to see:

  • when the quote was received from the relay websocket
  • when it was normalized and persisted
  • when strategy decided
  • when the execute command was emitted
  • when executor received it
  • salt source and salt age
  • signing time
  • relay response time
  • total quote-to-relay-result time
  • final relay result or error message
  • eventual outcome status when available

Competitiveness truth

The operator must be able to answer:

  • which pair/direction/request-kind is producing response failures
  • whether failures cluster by quote age, notional bucket, relay latency, or route
  • whether accepted responses have materially different timing than failed responses
  • whether nBTC/USDC is losing because local path latency is too high, because relay responses are late, or because incoming quotes are already stale

Response policy truth

If the system skips a maker response because the quote is too old or otherwise blocked by response policy, the decision must be durable and visible as a strategy rejection/block reason such as maker_quote_too_old, including the measured age, configured max age, pair id, and config version.

Semantic invariants

  • Pair-native fields remain canonical; no BTC/EURe-only latency or policy path is reintroduced.
  • Missing DB config, missing pair config, missing timing prerequisites, or invalid policy values fail closed.
  • A skipped response is not a failed relay submission; lifecycle views must keep strategy/policy skips distinct from relay failures.
  • Relay acceptance remains submission evidence only, not settlement or realized profit.
  • Response policy must be pair-scoped and versioned with DB strategy config or another DB-owned config row.
  • Existing nBTC/EURe compatibility and 49 bps edge behavior must survive.
  • nBTC/USDC must remain observable even when the policy skips responses.

Definition of done

  • New maker timing fields are persisted on quote, decision, command, result, and/or derived lifecycle records without relying on logs.
  • Dashboard and API surfaces expose pair-scoped latency and competitiveness summaries.
  • Relay errors such as quote not found or already finished are visible with timing context.
  • A DB-backed response-age policy exists if supported by the measured data, and skipped responses are durable, visible, and tested.
  • The policy cannot loosen live trading behavior or invent pair defaults.
  • Tests cover timing propagation, dashboard summaries, policy skip behavior, old-row compatibility, and nBTC/USDC regressions.
  • Full npm test and dashboard build pass.
  • The result is deployed through the repo workflow and validated against live post-deploy timing data.

Validation evidence required

  • Unit tests for timing-field propagation from normalized quote to strategy decision, command, executor result, and lifecycle row.
  • Tests proving relay failure messages remain visible with quote age and stage timing.
  • Tests proving nBTC/USDC rows aggregate by pair, direction, request kind, and result code.
  • Tests proving response-age policy skips old quotes without calling the executor or relay.
  • Tests proving missing/invalid DB policy fails closed or preserves existing behavior according to the explicit config contract.
  • Dashboard tests proving the competitiveness view shows timing buckets and skip reasons without hardcoded EURe/BTC labels.
  • Static deployment/config tests proving no latency or pair policy env vars are added.
  • Full npm test.
  • Dashboard bundle build.
  • Live evidence after deploy comparing at least pre-policy timing distribution and post-policy skip/accept/failure distribution.

Failure conditions

  • nBTC/USDC quote failures still cannot be explained beyond a relay error string.
  • Timing metrics exist only in logs and cannot be recomputed from durable rows.
  • The dashboard merges strategy/policy skips with relay submission failures.
  • A latency policy applies globally when it should be pair-scoped.
  • Missing DB config causes live responses instead of blocked behavior.
  • The implementation relaxes edge, notional, inventory, arming, or pair enablement.
  • A service requires new active pair or latency env vars to behave correctly.
  • Existing nBTC/EURe behavior regresses.

Current real before this turn

  • Pair-native asset, pair, strategy, route, preflight, attribution, valuation, dashboard, alert, and ops paths have been implemented and deployed.
  • nBTC/USDC maker decisions and result rows are visible from DB pair and route config.
  • Executor timing is already persisted with salt, signing, relay, and total timings.
  • Verifier salt caching has removed normal per-command salt RPC latency from the hot path.
  • Live cache-hit executor totals are often below 100 ms, but quote not found or already finished remains common.
  • Durable settlement truth still depends on observed inventory/outcome evidence, not relay acceptance alone.

Deliberately not built by this proof

  • Fee-complete realized PnL.
  • Venue-native terminal fill ids if the current relay/status paths do not expose them.
  • Arbitrary multi-venue or cross-region execution.
  • General solver-competition modeling beyond observed quote/response/outcome data.