fb547f24d9
All checks were successful
deploy / deploy (push) Successful in 48s
Instead of fetching up to 50k inventory snapshots (~442MB), scope the query to only the time range of the current submission batch with a 15-minute buffer. For a typical 1-hour batch this drops from 50k rows to ~300 rows, well within the 1280Mi pod memory limit. The coalesced_at_idx on intent_inventory_snapshots covers the BETWEEN clause so this remains efficient. Proof: history-writer OOM kills from refreshQuoteOutcomes inventory fetch Assumptions: 15min buffer covers the attribution window for all submissions Still fake: heuristic gap outcomes may attribute trades imprecisely
156 lines
9.7 KiB
Markdown
156 lines
9.7 KiB
Markdown
# Implementation Proof: Maker response latency and quote competitiveness
|
|
|
|
Status: open
|
|
Opened: 2026-05-18
|
|
|
|
## Target outcome
|
|
The maker path must explain why nBTC/USDC quote responses are accepted, rejected by the relay, or never become settled trades.
|
|
|
|
After the pair-native turn, the local executor hot path is fast enough that blind optimization is no longer justified. This turn succeeds when the repository records an end-to-end timing waterfall for each responded quote, exposes pair/request-kind competitiveness metrics to the operator, and uses DB-backed response policy to skip quotes that are already too old or otherwise empirically unwinnable.
|
|
|
|
## Why this is the next architecture test
|
|
Live evidence after verifier salt caching showed cache-hit executor results around:
|
|
|
|
- `submission_failed`: p50 total near 45 ms, p90 near 55 ms, p50 salt near 0 ms
|
|
- `quote_response_ok`: p50 total near 86 ms, p90 near 141 ms, p50 salt near 0 ms
|
|
- occasional blocking salt refreshes still exist, but most local salt latency is gone
|
|
|
|
Despite that, nBTC/USDC still produces many relay errors such as `quote not found or already finished`, and accepted relay responses are not reliably becoming durable settlement evidence.
|
|
|
|
The next question is not "can the code answer quotes at all"; it is whether each response is competitive at the moment we answer it, and whether the bot should skip quotes whose measured age and route characteristics make them unlikely to fill.
|
|
|
|
## Scope
|
|
- Add durable, pair-native maker timing fields from quote ingress through strategy, command, executor signing, relay response, and outcome attribution.
|
|
- Preserve existing executor timing fields and extend them with quote-age and stage timing that can be aggregated across services.
|
|
- Classify relay responses and failures by:
|
|
- pair id
|
|
- source/destination assets
|
|
- request kind
|
|
- notional asset and notional bucket
|
|
- route id/reference price id
|
|
- quote age at decision, command, executor receipt, and relay response
|
|
- relay result code and exact relay error message
|
|
- eventual outcome status when available
|
|
- Add operator-visible competitiveness metrics for nBTC/EURe and nBTC/USDC:
|
|
- response counts
|
|
- accepted response rate
|
|
- stale/already-finished response rate
|
|
- p50/p90/p99 stage latency
|
|
- quote age bucket outcomes
|
|
- pair/direction/request-kind breakdowns
|
|
- Add a DB-backed maker response-age policy only if evidence from the new timing data supports it.
|
|
- Keep response-age policy pair-scoped and fail closed:
|
|
- missing pair config must not create a default live responder
|
|
- missing or invalid max-age config must not loosen behavior
|
|
- age cutoffs may skip responses but must not reduce edge or raise notional limits
|
|
- Preserve the existing nBTC/EURe behavior unless the operator explicitly configures a stricter pair policy.
|
|
- Keep nBTC/USDC as the key regression pair.
|
|
- Deploy only through the repo workflow.
|
|
|
|
## Assumptions
|
|
- NEAR Intents quote availability can expire or be consumed independently of our local execution speed.
|
|
- The relay error `quote not found or already finished` is ambiguous; it may mean quote expiry, competing solver completion, or a relay-side state transition.
|
|
- Wall-clock timestamps across pods are good enough for operator timing ranges, but in-process monotonic timings remain the source of truth for sub-step latency.
|
|
- The active proof may reduce responses to empirically stale quotes, but it must not increase live-funds exposure, notional size, edge looseness, or automatic pair enablement.
|
|
- Postgres remains the canonical runtime source for pair and strategy config.
|
|
|
|
## Turn-shaping rules
|
|
- Do not add new env vars for latency policy, pair selection, or trade sizing.
|
|
- Do not relax edge thresholds, notional limits, inventory checks, arming, or pair enablement.
|
|
- Do not create new live trading by import or by default policy rows.
|
|
- Do not treat relay acceptance as a completed trade.
|
|
- Do not claim quote competitiveness from sampled logs alone; persist the fields needed to recompute it.
|
|
- Do not do manual deployment or cluster repair outside the repo workflow.
|
|
- If a policy change skips responses, the dashboard must show the skip reason explicitly.
|
|
|
|
## Non-goals
|
|
- No new execution venue.
|
|
- No geographic relocation, paid low-latency infrastructure, or permanent infra spend.
|
|
- No multi-replica executor race strategy.
|
|
- No response flooding or duplicate submissions.
|
|
- No automatic edge reduction to win more quotes.
|
|
- No fee-complete realized PnL.
|
|
- No venue-native terminal fill integration beyond existing evidence paths.
|
|
|
|
## Required operator behavior
|
|
|
|
### Quote timing truth
|
|
For a quote that reaches strategy and executor, the operator must be able to see:
|
|
|
|
- when the quote was received from the relay websocket
|
|
- when it was normalized and persisted
|
|
- when strategy decided
|
|
- when the execute command was emitted
|
|
- when executor received it
|
|
- salt source and salt age
|
|
- signing time
|
|
- relay response time
|
|
- total quote-to-relay-result time
|
|
- final relay result or error message
|
|
- eventual outcome status when available
|
|
|
|
### Competitiveness truth
|
|
The operator must be able to answer:
|
|
|
|
- which pair/direction/request-kind is producing response failures
|
|
- whether failures cluster by quote age, notional bucket, relay latency, or route
|
|
- whether accepted responses have materially different timing than failed responses
|
|
- whether nBTC/USDC is losing because local path latency is too high, because relay responses are late, or because incoming quotes are already stale
|
|
|
|
### Response policy truth
|
|
If the system skips a maker response because the quote is too old or otherwise blocked by response policy, the decision must be durable and visible as a strategy rejection/block reason such as `maker_quote_too_old`, including the measured age, configured max age, pair id, and config version.
|
|
|
|
## Semantic invariants
|
|
- Pair-native fields remain canonical; no BTC/EURe-only latency or policy path is reintroduced.
|
|
- Missing DB config, missing pair config, missing timing prerequisites, or invalid policy values fail closed.
|
|
- A skipped response is not a failed relay submission; lifecycle views must keep strategy/policy skips distinct from relay failures.
|
|
- Relay acceptance remains submission evidence only, not settlement or realized profit.
|
|
- Response policy must be pair-scoped and versioned with DB strategy config or another DB-owned config row.
|
|
- Existing nBTC/EURe compatibility and 49 bps edge behavior must survive.
|
|
- nBTC/USDC must remain observable even when the policy skips responses.
|
|
|
|
## Definition of done
|
|
- New maker timing fields are persisted on quote, decision, command, result, and/or derived lifecycle records without relying on logs.
|
|
- Dashboard and API surfaces expose pair-scoped latency and competitiveness summaries.
|
|
- Relay errors such as `quote not found or already finished` are visible with timing context.
|
|
- A DB-backed response-age policy exists if supported by the measured data, and skipped responses are durable, visible, and tested.
|
|
- The policy cannot loosen live trading behavior or invent pair defaults.
|
|
- Tests cover timing propagation, dashboard summaries, policy skip behavior, old-row compatibility, and nBTC/USDC regressions.
|
|
- Full `npm test` and dashboard build pass.
|
|
- The result is deployed through the repo workflow and validated against live post-deploy timing data.
|
|
|
|
## Validation evidence required
|
|
- Unit tests for timing-field propagation from normalized quote to strategy decision, command, executor result, and lifecycle row.
|
|
- Tests proving relay failure messages remain visible with quote age and stage timing.
|
|
- Tests proving nBTC/USDC rows aggregate by pair, direction, request kind, and result code.
|
|
- Tests proving response-age policy skips old quotes without calling the executor or relay.
|
|
- Tests proving missing/invalid DB policy fails closed or preserves existing behavior according to the explicit config contract.
|
|
- Dashboard tests proving the competitiveness view shows timing buckets and skip reasons without hardcoded EURe/BTC labels.
|
|
- Static deployment/config tests proving no latency or pair policy env vars are added.
|
|
- Full `npm test`.
|
|
- Dashboard bundle build.
|
|
- Live evidence after deploy comparing at least pre-policy timing distribution and post-policy skip/accept/failure distribution.
|
|
|
|
## Failure conditions
|
|
- nBTC/USDC quote failures still cannot be explained beyond a relay error string.
|
|
- Timing metrics exist only in logs and cannot be recomputed from durable rows.
|
|
- The dashboard merges strategy/policy skips with relay submission failures.
|
|
- A latency policy applies globally when it should be pair-scoped.
|
|
- Missing DB config causes live responses instead of blocked behavior.
|
|
- The implementation relaxes edge, notional, inventory, arming, or pair enablement.
|
|
- A service requires new active pair or latency env vars to behave correctly.
|
|
- Existing nBTC/EURe behavior regresses.
|
|
|
|
## Current real before this turn
|
|
- Pair-native asset, pair, strategy, route, preflight, attribution, valuation, dashboard, alert, and ops paths have been implemented and deployed.
|
|
- nBTC/USDC maker decisions and result rows are visible from DB pair and route config.
|
|
- Executor timing is already persisted with salt, signing, relay, and total timings.
|
|
- Verifier salt caching has removed normal per-command salt RPC latency from the hot path.
|
|
- Live cache-hit executor totals are often below 100 ms, but `quote not found or already finished` remains common.
|
|
- Durable settlement truth still depends on observed inventory/outcome evidence, not relay acceptance alone.
|
|
|
|
## Deliberately not built by this proof
|
|
- Fee-complete realized PnL.
|
|
- Venue-native terminal fill ids if the current relay/status paths do not expose them.
|
|
- Arbitrary multi-venue or cross-region execution.
|
|
- General solver-competition modeling beyond observed quote/response/outcome data.
|