Compare commits
No commits in common. "7ddefb500ead0053e6ac486616c7d58c319d9e43" and "f5ee95b325151b67670e2ef16a3a26b5c428e9ce" have entirely different histories.
7ddefb500e
...
f5ee95b325
10 changed files with 331 additions and 863 deletions
11
ARCHIVE.md
11
ARCHIVE.md
|
|
@ -8,20 +8,9 @@ Legacy note:
|
|||
## Implementation Turns
|
||||
|
||||
- 2026-04-02: `first-non-mocked-tradeable-loop-for-one-pair` closed with status `passed`. A live active-pair quote flowed through funding, inventory sync, strategy, and real mainnet quote responses with durable history.
|
||||
- 2026-04-04: `pre-credit-funding-visibility-and-operator-alerts` closed with status `passed`. Live BTC funding visibility, durable operator alerts, and rollout-safe armed-state persistence were proven on k3s without loosening spendable truth.
|
||||
- 2026-04-07: `operator-dashboard-foundation-funds-desk-and-operator-controls` closed with status `passed`. A real operator dashboard shipped with authenticated REST and WebSocket surfaces, truthful funded-capital profitability, live quote and trade visibility, and safe operator controls over the BTC/EURe system.
|
||||
- 2026-04-08: `cow-protocol-intent-based-venue-integration` closed with status `paused`. The CoW venue integration turn is paused after cluster investigation showed a real NEAR Intents ingest outage exposed a larger observability gap: the dashboard did not escalate stale quote flow or disconnected websocket clients, so runtime health and alerting need to be strengthened before adding a second venue.
|
||||
- 2026-04-08: `runtime-health-sentinel-alert-routing-and-anomaly-detection` closed with status `passed`. Runtime health is now sentinel-owned, stale truth no longer renders healthy, alert delivery and safe containment exist, and deployment automation rolls all repo-owned services from push.
|
||||
## Research Turns
|
||||
|
||||
## Planning Events
|
||||
- 2026-04-01: workflow files initialized for thesis, implementation proof, backlog, archive, and research lane.
|
||||
- 2026-04-01: active implementation proof rewritten from durable-history scaffolding to the first executable trade loop for one pair.
|
||||
- 2026-04-02: opened implementation turn `pre-credit-funding-visibility-and-operator-alerts` from backlog items O003, O004.
|
||||
- 2026-04-04: opened implementation turn `operator-dashboard-foundation-funds-desk-and-operator-controls` from backlog items I010, I011, I014.
|
||||
- 2026-04-04: revised the operator dashboard turn to require backend WebSockets, auth scaffolding from day one, and a narrow live quote plus successful-trade activity surface without polling.
|
||||
- 2026-04-04: revised the operator dashboard turn again to make profitability the primary operator question, with explicit deposit-baseline and simple-hold views plus attribution caveats where data is still incomplete.
|
||||
- 2026-04-04: added follow-on backlog items for treasury cashflow and fee truth, per-trade realized attribution, and quote-to-outcome markout analytics to support later profit analysis turns.
|
||||
- 2026-04-07: opened implementation turn `cow-protocol-intent-based-venue-integration` from backlog items I019.
|
||||
- 2026-04-08: opened implementation turn `runtime-health-sentinel-alert-routing-and-anomaly-detection` from backlog items I020.
|
||||
- 2026-04-08: opened implementation turn `quote-lifecycle-truth-and-execution-explanation` from backlog items I021.
|
||||
|
|
|
|||
|
|
@ -17,13 +17,7 @@ Rules:
|
|||
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
|
||||
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
|
||||
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
|
||||
- [I012] Quotes and execution analytics workbench: live quote tape, quote count and volume windows, size-bucket distributions, oracle-deviation views, per-quote decision and execution trace, and matched-only filters.
|
||||
- [I013] Benchmark and PnL analytics: compare trade PnL versus mark-to-market, all-BTC hold, all-EURe hold, passive 50/50 hold, and hindsight trade quality against later benchmarks.
|
||||
|
||||
- [I015] (soon) Benchmark-aware strategy thresholds and inventory-skewed fees: compare live inventory against deposit-time hold and target BTC/EURe mix before quoting away preferred inventory. tags=strategy,inventory,pnl
|
||||
- [I016] (soon) Treasury cashflow and fee ledger: durably record deposits, withdrawals, bridge or network costs, and other non-trade cashflows so dashboard profit can move from mark-to-market to true net PnL. tags=pnl,fees,treasury
|
||||
- [I017] (soon) Per-trade realized attribution: link each successful trade to actual settled inventory deltas and attributable costs so the system can report realized net contribution per trade instead of only expected edge. tags=trades,pnl,settlement
|
||||
- [I018] (later) Quote-to-outcome and markout analytics: link quote, response, decision, execution result, and settlement, then store later reference-price markouts to measure adverse selection and quote quality. tags=quotes,analytics,markout
|
||||
## Research Candidates
|
||||
- [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair.
|
||||
- [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold.
|
||||
|
|
|
|||
|
|
@ -1,251 +1,235 @@
|
|||
# Implementation Turn: quote lifecycle truth and execution explanation
|
||||
# Implementation Turn: pre-credit funding visibility and operator alerts
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-09
|
||||
Opened: 2026-04-02
|
||||
|
||||
## Goal
|
||||
Replace ambiguous quote and decision wording with a truthful per-quote lifecycle that tells the operator exactly why a quote was filtered, rejected, blocked, submitted, failed, not filled, or completed.
|
||||
Make the already-live BTC/EURe loop operationally understandable between external funding and spendable credit, and make critical stale or failed states queryable as durable alert records instead of transient logs.
|
||||
|
||||
## Selected backlog items
|
||||
- [I021] Quote lifecycle truth and execution explanation: replace ambiguous dashboard verdicts with a per-quote state machine that shows exactly why a quote was filtered, rejected, blocked, submitted, not filled, or executed, with durable reason codes and operator-facing traceability.
|
||||
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
|
||||
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
|
||||
|
||||
## Design rules
|
||||
- Treat quote lifecycle as product truth, not UI decoration.
|
||||
- Strategy verdict is not the final operator answer.
|
||||
- Prefer one explicit lifecycle derivation path shared by backend and dashboard over ad hoc page-specific wording.
|
||||
- Do not invent downstream certainty where durable evidence is absent.
|
||||
- Remove `Actionable` completely from operator-facing copy.
|
||||
## Design rule
|
||||
Keep the already-proven execution and inventory truth path intact:
|
||||
- verifier and bridge credit remain the only spendable truth
|
||||
- pre-credit visibility is additive observability
|
||||
- alerts are additive operability
|
||||
|
||||
## Problem statement for this turn
|
||||
The current dashboard still forces operators to infer too much:
|
||||
- `Actionable` does not say whether a command was emitted or submitted
|
||||
- the Strategy page mixes strategy and execution truth
|
||||
- executor-rejected rows are not clearly distinguishable from strategy-rejected rows
|
||||
- quote ids are truncated and awkward to use during debugging
|
||||
## Event backbone
|
||||
Retain Kafka as the backbone. Add only the minimal new topics required:
|
||||
|
||||
The repo already stores enough of the real lifecycle to do better:
|
||||
- quote id
|
||||
- decision id
|
||||
- emitted command id
|
||||
- execution result status and result code
|
||||
- `ops.funding_observation`
|
||||
- `ops.alert`
|
||||
|
||||
The turn therefore needs to improve:
|
||||
- lifecycle derivation
|
||||
- durable reason mapping
|
||||
- recent-row rendering
|
||||
- trace affordances
|
||||
These topics are append-only evidence streams, not control planes.
|
||||
|
||||
## Lifecycle model for this turn
|
||||
Implement one repo-owned lifecycle derivation for recent rows, using durable evidence in this order:
|
||||
## Durable store
|
||||
Extend PostgreSQL with new append-only families:
|
||||
- `funding_observations`
|
||||
- `ops_alerts`
|
||||
|
||||
1. Quote observed
|
||||
2. Strategy evaluated
|
||||
3. Command emitted or not emitted
|
||||
4. Executor result observed or absent
|
||||
5. Venue downstream outcome when available
|
||||
If a current-state materialization is needed, derive it from append-only records or keep it as a clearly named snapshot table. Do not replace the append-only record.
|
||||
|
||||
The first mandatory states are:
|
||||
- `Filtered`
|
||||
- `Rejected`
|
||||
- `Blocked`
|
||||
- `Submitted`
|
||||
- `Failed`
|
||||
- `Awaiting outcome`
|
||||
- `Completed`
|
||||
## Service changes
|
||||
|
||||
Suggested meanings:
|
||||
- `Filtered`
|
||||
quote never entered the active trade path or was excluded before strategy decision
|
||||
- `Rejected`
|
||||
strategy evaluated the quote and decided not to trade
|
||||
- `Blocked`
|
||||
strategy approved or emitted a command, but execution did not proceed due to control state or another repo-owned gate
|
||||
- `Submitted`
|
||||
executor accepted the command and successfully submitted a quote response
|
||||
- `Failed`
|
||||
execution submission failed technically
|
||||
- `Awaiting outcome`
|
||||
submitted to venue, but no later durable terminal venue outcome exists yet
|
||||
- `Completed`
|
||||
durable evidence shows the trade completed successfully
|
||||
### 1. `liquidity-manager`
|
||||
Extend the existing treasury owner instead of inventing a broad new funding stack.
|
||||
|
||||
Do not show states we cannot support yet for a given row.
|
||||
New responsibilities:
|
||||
- retain active funding handles by chain and asset
|
||||
- poll configured chain observers for those handles
|
||||
- emit `ops.funding_observation`
|
||||
- correlate chain observations with bridge `recent_deposits`
|
||||
- expose latest observations and credit-correlation state through `/state`
|
||||
|
||||
## Reason-code model
|
||||
For each lifecycle state, map durable payload fields to a small operator reason taxonomy.
|
||||
Expected state shape additions:
|
||||
- `funding_observations_by_handle`
|
||||
- `latest_funding_observation_at`
|
||||
- `uncredited_funding_total_by_asset`
|
||||
- `credit_correlation`
|
||||
|
||||
Examples:
|
||||
- strategy reason codes:
|
||||
- unsupported_pair
|
||||
- below_edge_threshold
|
||||
- inventory_unavailable
|
||||
- stale_reference_price
|
||||
- executor reason codes:
|
||||
- executor_disarmed
|
||||
- executor_paused
|
||||
- submission_failed
|
||||
- quote_response_ok
|
||||
- downstream outcome reasons if available:
|
||||
- expired
|
||||
- not_filled
|
||||
- completed
|
||||
Control additions:
|
||||
- `POST /refresh-funding-observations`
|
||||
- optional `POST /pause-funding-observer`
|
||||
- optional `POST /resume-funding-observer`
|
||||
|
||||
If the exact reason is missing:
|
||||
- expose `reason_unknown`
|
||||
- keep the row truthful instead of synthesizing an explanation
|
||||
Important implementation constraints:
|
||||
- do not change withdrawal behavior
|
||||
- do not reuse spendable inventory fields for pre-credit state
|
||||
- keep BTC and EURe observation records in one shared schema
|
||||
|
||||
## Backend changes
|
||||
### 2. `inventory-sync`
|
||||
Keep current spendable accounting intact.
|
||||
|
||||
### 1. Add a lifecycle derivation helper
|
||||
Create or extend a backend module that derives quote lifecycle from:
|
||||
- recent trade decisions
|
||||
- recent execution results
|
||||
- successful trade records
|
||||
- any available quote-status or venue result surfaces
|
||||
Possible additions:
|
||||
- read the latest funding observations
|
||||
- expose a separate `pre_credit_inbound` or `funding_visibility` field in `/state`
|
||||
|
||||
It should emit a normalized row object with:
|
||||
- `quote_id`
|
||||
- `decision_id`
|
||||
- `command_id`
|
||||
- `pair`
|
||||
- `direction`
|
||||
- `lifecycle_state`
|
||||
- `lifecycle_label`
|
||||
- `reason_code`
|
||||
- `reason_text`
|
||||
- timestamps for the latest known stage
|
||||
- stage details for tooltips or drilldown
|
||||
Hard rule:
|
||||
- `spendable`, `pending_inbound`, and strategy-facing credited truth must not become looser
|
||||
|
||||
### 2. Join decision and execution truth explicitly
|
||||
The backend should no longer leave the frontend to infer execution from isolated tables.
|
||||
### 3. `history-writer`
|
||||
Consume and persist the new topics:
|
||||
- `ops.funding_observation`
|
||||
- `ops.alert`
|
||||
|
||||
For each recent quote/decision row:
|
||||
- attach the matching execution result by `command_id`, `decision_id`, or `quote_id`
|
||||
- attach successful-trade or later terminal evidence where available
|
||||
- expose whether the row is strategy-only, strategy-plus-command, or strategy-plus-execution
|
||||
Expose through `/state`:
|
||||
- latest funding-observation write time
|
||||
- latest alert write time
|
||||
- counts or offsets for the new topics
|
||||
|
||||
### 3. Preserve operator drilldown identifiers
|
||||
Ensure the bootstrap payload exposes:
|
||||
- full quote id
|
||||
- full decision id
|
||||
- full command id
|
||||
Add query-friendly indexes for:
|
||||
- `tx_hash`
|
||||
- `funding_handle`
|
||||
- `alert_code`
|
||||
- `ingested_at`
|
||||
|
||||
Avoid requiring the frontend to reconstruct or guess identifiers from formatted strings.
|
||||
### 4. `ops-sentinel` or equivalent alert evaluator
|
||||
Add one small service only if needed to keep alert logic separate and testable.
|
||||
|
||||
## Dashboard changes
|
||||
Responsibilities:
|
||||
- consume:
|
||||
- `ref.market_price`
|
||||
- `state.intent_inventory`
|
||||
- `ops.liquidity_action`
|
||||
- `ops.funding_observation`
|
||||
- `exec.trade_result`
|
||||
- evaluate policy windows for stale and stuck conditions
|
||||
- emit `ops.alert` raise/clear transitions
|
||||
- expose current alert state
|
||||
|
||||
### 4. Remove forbidden language
|
||||
Remove `Actionable` from:
|
||||
- Strategy page tables
|
||||
- any lifecycle badge or verdict cell
|
||||
- any supporting labels or legends
|
||||
Preferred alert model:
|
||||
- stable `alert_code`
|
||||
- `status`: `raised` or `cleared`
|
||||
- `severity`
|
||||
- `reason`
|
||||
- `first_raised_at`
|
||||
- `last_evaluated_at`
|
||||
- correlation IDs when available
|
||||
|
||||
Replace it with explicit state labels driven by lifecycle derivation.
|
||||
Minimal alert set for this turn:
|
||||
- `reference_price_stale`
|
||||
- `inventory_snapshot_stale`
|
||||
- `funding_seen_unconfirmed`
|
||||
- `funding_confirmed_credit_pending`
|
||||
- `funding_stuck`
|
||||
- `executor_submission_failed`
|
||||
|
||||
### 5. Make recent rows self-explanatory
|
||||
For each row, render:
|
||||
- primary lifecycle state
|
||||
- secondary reason text
|
||||
- quote id with copy action
|
||||
- command id if emitted
|
||||
- timestamps
|
||||
Do not add Slack, email, or paging integrations in this turn unless required to prove the path. Durable alert records plus HTTP state are sufficient.
|
||||
|
||||
The operator should be able to scan rows and answer:
|
||||
- why no trade happened
|
||||
- whether the system tried to trade
|
||||
- whether failure was strategic, operational, or downstream
|
||||
## Chain observer plan
|
||||
|
||||
### 6. Add trace affordances
|
||||
At minimum:
|
||||
- copy button for quote id
|
||||
- avoid over-truncating ids without recovery path
|
||||
- show linked ids in a dedicated trace column or expanded detail panel
|
||||
### BTC
|
||||
Must be the first-class proof path.
|
||||
|
||||
If the row layout gets crowded, prefer an expandable detail tray over hiding identifiers.
|
||||
Implementation expectations:
|
||||
- configurable observer endpoint
|
||||
- look up the configured BTC deposit address
|
||||
- detect:
|
||||
- mempool appearance when available
|
||||
- confirmation count
|
||||
- credited transition once bridge/verifier catches up
|
||||
|
||||
## Page-level application
|
||||
Assumption to keep explicit in code:
|
||||
- a chain observer can disappear or lag independently of the bridge
|
||||
|
||||
### Strategy page
|
||||
This page should become the primary recent quote-decision-execution lifecycle surface.
|
||||
### Gnosis / EURe
|
||||
Nice-to-have within the same schema, but BTC is the proof-critical path.
|
||||
|
||||
It should show:
|
||||
- the latest recent rows for the active pair
|
||||
- lifecycle state rather than strategy-only verdict
|
||||
- explicit explanation text
|
||||
If included this turn:
|
||||
- watch the configured deposit address for EURe token transfers
|
||||
- represent observation state in the same event model
|
||||
|
||||
If a strategy-only summary remains, it must be visually separate from per-quote lifecycle truth.
|
||||
## Record shapes
|
||||
|
||||
### Related quote surfaces
|
||||
Inspect quote and system surfaces for similar ambiguity and align the wording if they expose the same concepts.
|
||||
### `ops.funding_observation`
|
||||
Required fields:
|
||||
- `funding_observation_id`
|
||||
- `account_id`
|
||||
- `asset_id`
|
||||
- `chain`
|
||||
- `funding_handle`
|
||||
- `source`
|
||||
- `tx_hash`
|
||||
- `status`
|
||||
- `amount`
|
||||
- `confirmations`
|
||||
- `first_seen_at`
|
||||
- `last_seen_at`
|
||||
- `credited_at` when known
|
||||
- `bridge_deposit_tx_hash` when correlated
|
||||
|
||||
Do not let one page say `Submitted` while another page still says `Actionable` for the same row.
|
||||
### `ops.alert`
|
||||
Required fields:
|
||||
- `alert_event_id`
|
||||
- `alert_code`
|
||||
- `status`
|
||||
- `severity`
|
||||
- `reason`
|
||||
- `service_scope`
|
||||
- `pair` when relevant
|
||||
- `asset_id` when relevant
|
||||
- `tx_hash` when relevant
|
||||
- `raised_at`
|
||||
- `cleared_at`
|
||||
- `details`
|
||||
|
||||
## Data and state edge cases
|
||||
- Strategy decision exists, no command emitted:
|
||||
render as `Rejected` with strategy reason
|
||||
- Command emitted, no execution result yet:
|
||||
render as `Blocked` or `Awaiting executor` only if that distinction is durably supportable; otherwise use a truthful pending label
|
||||
- Execution result `executor_disarmed`:
|
||||
render as `Blocked` with reason `executor disarmed`
|
||||
- Execution result `submission_failed`:
|
||||
render as `Failed`
|
||||
- Execution result `submitted`:
|
||||
render as `Submitted` or `Awaiting outcome`
|
||||
- Successful trade summary exists but no explicit per-quote completion event:
|
||||
only promote to `Completed` where the durable linkage is real
|
||||
## Control surface expectations
|
||||
|
||||
## Concrete implementation order
|
||||
### `liquidity-manager`
|
||||
Must expose:
|
||||
- active deposit handles
|
||||
- latest pre-credit funding observations
|
||||
- latest credit correlation
|
||||
- whether the funding observer is healthy or paused
|
||||
|
||||
### Phase 1. Define lifecycle derivation
|
||||
- inspect current durable decision and execution payloads
|
||||
- write the normalized lifecycle state mapping
|
||||
- define forbidden and allowed operator labels
|
||||
### alert evaluator
|
||||
Must expose:
|
||||
- current active alerts
|
||||
- latest cleared alerts
|
||||
- per-alert evaluation timestamps
|
||||
- pause state
|
||||
|
||||
### Phase 2. Implement backend aggregation
|
||||
- derive unified recent lifecycle rows
|
||||
- expose full identifiers and reason codes
|
||||
- keep old consumers working until the frontend is switched
|
||||
### `history-writer`
|
||||
Must expose the new topic offsets and write status for funding observations and alerts.
|
||||
|
||||
### Phase 3. Update Strategy page rendering
|
||||
- replace verdict column with lifecycle state
|
||||
- add reason text
|
||||
- add quote-id copy affordance
|
||||
- surface command id and execution state where relevant
|
||||
## Tests
|
||||
Required automated coverage:
|
||||
- BTC funding observation remains non-spendable before credit
|
||||
- alert transitions raise then clear on recovered stale state
|
||||
- funding observation correlates to a later credited deposit without losing the original tx hash
|
||||
- executor failure produces an alert event
|
||||
|
||||
### Phase 4. Tighten wording and consistency
|
||||
- remove `Actionable`
|
||||
- align supporting labels
|
||||
- ensure blocked vs rejected vs submitted are clearly distinct
|
||||
If a meaningful automated test cannot be written for a subpath, stop and record why instead of hand-waving.
|
||||
|
||||
### Phase 5. Validate with live recent rows
|
||||
- verify a row rejected due to executor disarmed renders as blocked with reason
|
||||
- verify a submitted row renders as submitted
|
||||
- verify quote ids can be copied and used for tracing
|
||||
## Validation plan
|
||||
- Safe induced stale-price alert:
|
||||
- pause `market-reference-ingest`
|
||||
- wait past freshness window
|
||||
- observe `reference_price_stale`
|
||||
- resume and observe clear
|
||||
- Safe induced stale-inventory alert:
|
||||
- pause `inventory-sync`
|
||||
- wait past freshness window
|
||||
- observe `inventory_snapshot_stale`
|
||||
- resume and observe clear
|
||||
- Funding visibility proof:
|
||||
- use a real deposit address
|
||||
- observe pre-credit chain state before bridge credit where timing allows
|
||||
- later observe credit correlation
|
||||
- Executor failure alert proof:
|
||||
- use a controlled non-destructive failure mode such as temporary relay endpoint override in a safe environment or a replayable failure fixture
|
||||
- verify `executor_submission_failed`
|
||||
|
||||
## Test plan
|
||||
- unit tests for lifecycle derivation from:
|
||||
- strategy-rejected rows
|
||||
- executor-disarmed rows
|
||||
- submission-failed rows
|
||||
- submitted rows
|
||||
- dashboard bootstrap tests for:
|
||||
- forbidden `Actionable` removal
|
||||
- explicit lifecycle labels
|
||||
- reason text rendering
|
||||
- identifier exposure
|
||||
- frontend component tests if needed for copy affordance or row rendering logic
|
||||
## Out of scope on purpose
|
||||
- No new trading strategy
|
||||
- No historical backtest engine
|
||||
- No broad observability stack
|
||||
- No polished dashboard frontend
|
||||
- No automated treasury refills
|
||||
|
||||
No lifecycle ambiguity fix is complete without a regression test proving the old ambiguous wording cannot return.
|
||||
|
||||
## Validation checklist against the proof
|
||||
- `Actionable` no longer appears
|
||||
- strategy approval is visibly distinct from execution submission
|
||||
- recent blocked rows explain why they did not trade
|
||||
- recent submitted rows show that they were submitted
|
||||
- quote ids are directly usable from the dashboard
|
||||
|
||||
## Failure modes to plan for
|
||||
- the backend joins rows incorrectly and attributes the wrong execution result
|
||||
- the UI uses softer wording than the backend lifecycle state
|
||||
- older rows lack enough evidence and the UI pretends certainty
|
||||
- ids are still truncated without a copy or expand path
|
||||
## Still fake at turn open
|
||||
- Pre-credit funding visibility is still missing from the live cluster.
|
||||
- Alert state is still mostly implicit in service logs and manual inspection.
|
||||
- There is no durable operator-facing record yet for "funds are on the way but not spendable."
|
||||
|
|
|
|||
256
PROOF.md
256
PROOF.md
|
|
@ -1,154 +1,154 @@
|
|||
# Implementation Proof: quote lifecycle truth and execution explanation
|
||||
# Implementation Proof: pre-credit funding visibility and operator alerts
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-09
|
||||
Opened: 2026-04-02
|
||||
|
||||
## Target outcome
|
||||
This turn proves that `unrip` can explain every quote outcome in operator-visible, durable terms instead of forcing the operator to infer meaning from ambiguous labels.
|
||||
The next turn is complete only when `unrip` can show operators the gap between "funds sent" and "funds spendable" with durable evidence, and can surface actionable alert state for the live loop without requiring log-diving or manual SQL every time something stalls.
|
||||
|
||||
The concrete target is the live NEAR Intents BTC/EURe system:
|
||||
- each quote row must expose its current lifecycle state
|
||||
- each non-trade outcome must expose the decisive reason code
|
||||
- execution submission must be distinguishable from strategy approval
|
||||
- blocked, rejected, submitted, failed, and not-filled paths must be visibly different
|
||||
- quote identifiers must be directly usable by operators for tracing and support
|
||||
|
||||
## Why this is a meaningful architecture test
|
||||
The current operator surface still fails a core thesis requirement:
|
||||
- the Strategy page shows `Actionable`, which does not tell the operator whether a trade was actually submitted
|
||||
- an operator looking at one quote cannot answer, at a glance, why it did or did not become a trade
|
||||
- quote ids are hidden behind truncation with no direct copy affordance
|
||||
- execution truth exists durably in PostgreSQL, but the UI does not surface the lifecycle coherently
|
||||
|
||||
That is not just a copy problem. It is an observability gap in the trading product itself. If the system cannot explain a quote outcome precisely, execution is outrunning observability.
|
||||
This turn does not expand the trade hot path. It makes the existing live system more explainable and more operable.
|
||||
|
||||
## Hypothesis
|
||||
`unrip` becomes more trustworthy if quote handling is modeled and rendered as an explicit lifecycle instead of a single strategy verdict:
|
||||
- strategy evaluation is only one stage in the lifecycle
|
||||
- executor acceptance or rejection must be first-class state
|
||||
- venue submission and downstream outcome must be represented separately
|
||||
- durable reason codes must drive operator labels
|
||||
- forbidden labels that collapse multiple meanings, especially `Actionable`, must be removed
|
||||
`unrip` becomes materially safer to operate once it can:
|
||||
|
||||
The turn passes only if an operator can inspect a quote and immediately understand whether it was filtered, rejected, blocked, submitted, failed, awaiting venue outcome, not filled, or completed, and why.
|
||||
1. observe configured funding handles before NEAR Intents credit
|
||||
2. persist chain-level funding observations separately from spendable inventory
|
||||
3. link pre-credit observations to later bridge and verifier credit where possible
|
||||
4. emit durable alert state for stale prices, stale inventory, stuck funding, and failed execution submissions
|
||||
5. expose that state through the same small control surfaces and PostgreSQL audit trail as the rest of the system
|
||||
|
||||
If pre-credit funding remains invisible, or alert state still lives only in transient logs, the live loop is still too opaque for routine funded operation.
|
||||
|
||||
## Scope
|
||||
- [I021] Quote lifecycle truth and execution explanation: replace ambiguous dashboard verdicts with a per-quote state machine that shows exactly why a quote was filtered, rejected, blocked, submitted, not filled, or executed, with durable reason codes and operator-facing traceability.
|
||||
- Cover the live path from:
|
||||
- raw or normalized quote observation
|
||||
- strategy decision
|
||||
- execute-trade command emission
|
||||
- executor result
|
||||
- solver relay or venue outcome where available
|
||||
- Update the dashboard surfaces that currently expose quote or strategy truth:
|
||||
- Strategy page
|
||||
- any quote or recent-decision tables tied to the active pair
|
||||
- quote-id presentation and operator trace affordances
|
||||
|
||||
## Assumptions
|
||||
- The existing durable stores already contain enough information for at least the current live path through strategy decision and executor result.
|
||||
- Some downstream venue-outcome states may still be partially fake or unavailable for older rows; if so, the UI must say that plainly rather than implying more certainty.
|
||||
- The immediate turn should prioritize truthful lifecycle explanation over broader analytics such as markout or long-window outcome attribution.
|
||||
|
||||
## Turn-shaping rules
|
||||
- `Actionable` is forbidden as an operator-facing state or label.
|
||||
- Do not add a second analytics product. Stay focused on per-quote lifecycle truth for the live active pair.
|
||||
- Do not invent lifecycle states that cannot be backed by durable repo-owned evidence.
|
||||
- If a state transition is inferred rather than durably observed, the UI must make that distinction explicit.
|
||||
- Prefer a small, explicit state machine over a long list of loosely related badges.
|
||||
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
|
||||
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
|
||||
|
||||
## Non-goals
|
||||
- No new venue integrations.
|
||||
- No broad historical markout analytics turn.
|
||||
- No new execution automation or risk widening.
|
||||
- No redesign of the entire dashboard visual system beyond what is needed to make the quote lifecycle understandable.
|
||||
- No new venue, pair, or strategy logic.
|
||||
- No dashboard or polished UI.
|
||||
- No automatic treasury actions or auto-refunding.
|
||||
- No attempt to treat chain-level observations as spendable inventory.
|
||||
- No change to the live execution arming model.
|
||||
|
||||
## Required operator behavior
|
||||
## Source-of-truth rule
|
||||
Spendable inventory remains the existing truth:
|
||||
- bridge and verifier credit determine spendable balances
|
||||
- chain-level observations are visibility only
|
||||
|
||||
### Lifecycle truth
|
||||
For each visible quote or decision row, the operator must be able to identify the current lifecycle state from a bounded set such as:
|
||||
- `Filtered`
|
||||
- `Rejected by strategy`
|
||||
- `Blocked before submit`
|
||||
- `Submission failed`
|
||||
- `Submitted`
|
||||
- `Awaiting venue outcome`
|
||||
- `Not filled` or equivalent final non-fill state, if durable evidence exists
|
||||
- `Completed` or equivalent successful terminal state, if durable evidence exists
|
||||
The new pre-credit path must never be allowed to make a direction tradable earlier than the verifier does.
|
||||
|
||||
Exact labels may vary, but they must be specific and mutually meaningful.
|
||||
## Required runtime behavior
|
||||
|
||||
### Reason truth
|
||||
Each non-terminal or terminal non-trade state must expose a clear decisive reason, such as:
|
||||
- unsupported pair
|
||||
- below edge threshold
|
||||
- inventory unavailable
|
||||
- executor disarmed
|
||||
- executor paused
|
||||
- submission failed
|
||||
- venue timeout
|
||||
- quote expired
|
||||
### Funding visibility
|
||||
- The system must know the currently active funding handles for BTC and EURe.
|
||||
- For configured chains, it must watch those handles before NEAR Intents credit appears.
|
||||
- It must distinguish at least these states where applicable:
|
||||
- `SEEN_UNCONFIRMED`
|
||||
- `SEEN_CONFIRMED`
|
||||
- `CREDIT_PENDING`
|
||||
- `CREDITED`
|
||||
- `FAILED_OR_STUCK`
|
||||
- BTC is the must-prove chain because that is where live funding latency was operationally visible.
|
||||
- Gnosis support may share the same event model even if its confirmation behavior is simpler.
|
||||
|
||||
If the decisive reason is not known, the surface must say that plainly instead of inventing confidence.
|
||||
### Alerts
|
||||
- Alerts must be durable records, not only log lines.
|
||||
- At minimum the system must raise and clear alert state for:
|
||||
- stale reference price
|
||||
- stale inventory snapshot
|
||||
- funding seen but not credited within policy
|
||||
- execution submission failure
|
||||
- Alert transitions must be inspectable through HTTP state and PostgreSQL.
|
||||
|
||||
### Traceability
|
||||
For each quote row the operator must be able to access:
|
||||
- quote id in a non-hidden, copyable form
|
||||
- decision id where present
|
||||
- command id where present
|
||||
- execution result where present
|
||||
- pair and direction
|
||||
## Service expectations
|
||||
|
||||
The operator must be able to reason from a single row without manually cross-correlating multiple pages.
|
||||
### `liquidity-manager`
|
||||
Must become the owner of chain-level funding observations because it already owns deposit handles and treasury state.
|
||||
|
||||
### UI language
|
||||
The UI must not render `Actionable`.
|
||||
It must:
|
||||
- refresh and retain active funding handles
|
||||
- ingest chain-level funding observations
|
||||
- reconcile them against bridge deposit state
|
||||
- publish durable funding-observation records
|
||||
- expose current per-handle funding state
|
||||
|
||||
Any replacement label must answer a concrete operator question, such as:
|
||||
- did strategy approve this?
|
||||
- was a command emitted?
|
||||
- was the command blocked?
|
||||
- was it submitted?
|
||||
- did it fail?
|
||||
It must not:
|
||||
- mark funds spendable
|
||||
- trade on pre-credit observations
|
||||
|
||||
### `inventory-sync`
|
||||
May surface pre-credit funding context, but only under a clearly separate non-spendable field.
|
||||
|
||||
It must not:
|
||||
- merge pre-credit observations into `spendable`
|
||||
|
||||
### `history-writer`
|
||||
Must persist the new record families:
|
||||
- funding observations
|
||||
- alert events
|
||||
- optionally current alert snapshots if the implementation separates events from state
|
||||
|
||||
### Alert evaluator
|
||||
This may be a new small service or a tightly scoped extension of an existing one, but it must:
|
||||
- evaluate staleness and stuck conditions from durable inputs
|
||||
- emit durable alert events
|
||||
- expose current alert state and the latest reasons
|
||||
|
||||
No broad orchestration or dashboard service should be introduced just to satisfy this proof.
|
||||
|
||||
## Required durable storage
|
||||
PostgreSQL must store at least:
|
||||
- funding observations before bridge credit
|
||||
- alert events or alert snapshots
|
||||
- enough timestamps and IDs to correlate:
|
||||
- funding handle
|
||||
- chain tx hash
|
||||
- later bridge tx hash or deposit record
|
||||
- resulting verifier credit snapshot when available
|
||||
|
||||
Kafka remains the event backbone.
|
||||
|
||||
## Required control surface
|
||||
At minimum operators must be able to inspect:
|
||||
- active funding handles
|
||||
- latest pre-credit observations by handle
|
||||
- confirmation depth or equivalent chain state when available
|
||||
- whether a funding action is still pending credit
|
||||
- current active alerts and their reasons
|
||||
|
||||
If a new alert service exists, it must expose:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
|
||||
## Definition of done
|
||||
- `Actionable` is removed from operator-facing dashboard surfaces.
|
||||
- A durable quote lifecycle model exists in repo-owned code and is used by the dashboard.
|
||||
- At least the current live quote path through strategy decision and executor result is rendered coherently per quote.
|
||||
- The operator can tell, from one row, why a recent quote did or did not turn into a submitted trade.
|
||||
- Quote ids are copyable and clearly visible enough for tracing.
|
||||
- Regression tests cover at least:
|
||||
- strategy-approved but executor-disarmed rows
|
||||
- submitted rows
|
||||
- forbidden ambiguous label removal
|
||||
|
||||
For this turn to close with status `passed`, the specific operator question:
|
||||
|
||||
`Why did this quote not trade?`
|
||||
|
||||
must be answerable directly from the dashboard for recent rows without needing manual database inspection.
|
||||
|
||||
## Validation evidence required
|
||||
- direct UI or bootstrap evidence that recent rows show explicit lifecycle states instead of `Actionable`
|
||||
- direct evidence that a strategy-approved but executor-disarmed row renders as blocked or rejected with reason
|
||||
- direct evidence that a submitted row renders as submitted
|
||||
- direct evidence that quote ids are directly usable for tracing
|
||||
- automated test evidence for lifecycle derivation and dashboard rendering
|
||||
- The live cluster still runs the previously proven funded trade loop unchanged for spendable truth.
|
||||
- At least one real funding handle is watched at chain level before bridge credit.
|
||||
- For at least one real deposit path, the system records a pre-credit observation before or during confirmation and later records the credited state separately.
|
||||
- PostgreSQL contains durable records for the pre-credit funding path and alert path.
|
||||
- Operators can inspect current funding observations and current alerts through control APIs.
|
||||
- A stale price or stale inventory condition can be induced safely and becomes a durable alert.
|
||||
- A funding delay or manually injected stuck condition can be represented as a durable alert with explicit reason fields.
|
||||
- A failed execution submission path is represented as an alert without inventing fake venue traffic.
|
||||
- Tests cover:
|
||||
- pre-credit observations staying non-spendable
|
||||
- alert raise and clear transitions
|
||||
- correlation of funding observation to later credit when identifiers are available
|
||||
|
||||
## Failure conditions
|
||||
- `Actionable` still appears in the dashboard
|
||||
- the operator still cannot distinguish strategy approval from execution submission
|
||||
- non-trade rows still lack a decisive reason
|
||||
- quote ids remain hidden or non-copyable
|
||||
- lifecycle labels are only cosmetic and not backed by durable repo-owned state
|
||||
- Funding observations exist only in logs and are not queryable later.
|
||||
- Pre-credit observations leak into spendable inventory or strategy gating.
|
||||
- Alerts cannot be queried as current state.
|
||||
- The only proof of stuck funding is a human manually watching a block explorer.
|
||||
- The implementation adds a dashboard shell without stronger runtime truth.
|
||||
|
||||
## Current real before this turn
|
||||
- strategy decisions are stored durably
|
||||
- execution results are stored durably
|
||||
- command ids, decision ids, and quote ids already exist in the durable path
|
||||
- the operator dashboard already serves recent decisions and execution-adjacent state
|
||||
|
||||
## Deliberately not built by this proof
|
||||
- full venue settlement attribution for all historic trades
|
||||
- generalized quote analytics beyond lifecycle explanation
|
||||
- multi-venue lifecycle harmonization
|
||||
## Current real
|
||||
- The first funded BTC/EURe live loop is already proven:
|
||||
- real quote ingest
|
||||
- real reference pricing
|
||||
- real credited inventory
|
||||
- real strategy decisions
|
||||
- real `quote_response` submissions
|
||||
- durable event chain in PostgreSQL
|
||||
- Portfolio metrics are now durably computed and exposed, but alerting and pre-credit funding visibility are still incomplete.
|
||||
|
|
|
|||
|
|
@ -1,284 +0,0 @@
|
|||
# Implementation Turn: runtime health sentinel alert routing and anomaly detection
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-08
|
||||
|
||||
## Goal
|
||||
Make the live NEAR Intents system loudly and durably aware of broken runtime truth.
|
||||
|
||||
The system must detect stale or disconnected quote flow, grade service health from actual runtime behavior instead of mere reachability, surface that severity aggressively in the dashboard, emit durable alerts plus at least one external notification, and support tightly scoped safe containment for non-fund-moving failures.
|
||||
|
||||
## Selected backlog items
|
||||
- [I020] Runtime health sentinel, alert routing, and anomaly detection: detect stale or broken service behavior across ingest, execution, persistence, and operator surfaces; surface critical failures loudly in the dashboard; emit durable alerts plus external notifications; and support tightly scoped safe remediation for non-fund-moving failures.
|
||||
|
||||
## Design rules
|
||||
- Keep runtime health truth repo-owned and tied to the existing Kafka plus PostgreSQL pipeline.
|
||||
- Prefer deterministic invariants first. Add anomaly detection as a secondary layer for things we did not name ahead of time.
|
||||
- Do not trust service-local `/healthz` alone for end-to-end health.
|
||||
- Safe containment beats speculative auto-repair.
|
||||
- No new fund-moving actions or risk widening.
|
||||
|
||||
## Problem statement for this turn
|
||||
The live incident showed three separate weaknesses:
|
||||
- `near-intents-ingest` could remain running while quote truth stopped
|
||||
- the dashboard could display large freshness ages while still calling the service healthy
|
||||
- no durable alert or external notification made the issue impossible to miss
|
||||
|
||||
The turn therefore needs to improve:
|
||||
- runtime-state collection
|
||||
- cross-service health evaluation
|
||||
- durable alert generation
|
||||
- operator severity rendering
|
||||
- external alert delivery
|
||||
- bounded anomaly detection
|
||||
|
||||
## Minimal architecture changes for this turn
|
||||
- `ops-sentinel` becomes the runtime health authority, not just a narrow stale-data checker.
|
||||
- services expose richer state and truthful health hints through existing control APIs.
|
||||
- `operator-dashboard` consumes alert severity and derived health rather than inferring “healthy” from reachability.
|
||||
- one outbound notifier module sends raised or cleared alerts to an external webhook-style sink.
|
||||
- anomaly logic runs in repo-owned code over service state and durable or live counters rather than free-text logs.
|
||||
|
||||
## Architecture changes
|
||||
|
||||
### 1. Extend service-local state and health surfaces
|
||||
Each relevant service should expose enough state for sentinel evaluation.
|
||||
|
||||
For `near-intents-ingest`:
|
||||
- websocket connected flag
|
||||
- last websocket message time
|
||||
- last matching quote time
|
||||
- last published quote time
|
||||
- publish error count
|
||||
- pair filter state
|
||||
|
||||
For `trade-executor`:
|
||||
- solver relay connected flag
|
||||
- pending requests or in-flight count
|
||||
- last quote-status or relay message time
|
||||
- last error
|
||||
- armed and paused state
|
||||
|
||||
For `history-writer`:
|
||||
- last write time
|
||||
- last alert write time
|
||||
- offsets by topic
|
||||
- database connectivity
|
||||
- last error
|
||||
|
||||
For `operator-dashboard`:
|
||||
- recent upstream source errors
|
||||
- websocket fan-out status if available
|
||||
- backend bootstrap-source failures
|
||||
|
||||
Do not rely on Kubernetes liveness alone.
|
||||
|
||||
### 2. Expand `ops-sentinel` into runtime-health evaluation
|
||||
Add a service-state polling loop inside `ops-sentinel` for the existing control APIs.
|
||||
|
||||
The sentinel should evaluate at least:
|
||||
- service reachability
|
||||
- service-local freshness
|
||||
- disconnected websocket clients
|
||||
- pipeline stalls where source activity stops or downstream persistence stops
|
||||
- unsafe armed-state combinations
|
||||
- self-health staleness
|
||||
|
||||
Prefer deriving named condition objects first, then reconciling them into alert state.
|
||||
|
||||
### 3. Introduce a runtime health model
|
||||
Define explicit derived states per service:
|
||||
- `healthy`
|
||||
- `warning`
|
||||
- `critical`
|
||||
- `offline`
|
||||
- `paused`
|
||||
|
||||
Service severity must be derived from:
|
||||
- explicit `/healthz ok: false`
|
||||
- freshness thresholds
|
||||
- disconnected state
|
||||
- active alert severity for that service
|
||||
- pipeline mismatches tied to that service
|
||||
|
||||
This logic should live in repo-owned code shared or mirrored between sentinel and dashboard so the UI cannot drift into a softer interpretation.
|
||||
|
||||
### 4. Add new deterministic alert classes
|
||||
Extend `src/core/alert-engine.mjs` to cover at least:
|
||||
- ingest disconnected
|
||||
- ingest last quote stale
|
||||
- ingest last publish stale
|
||||
- executor relay disconnected
|
||||
- history writer stalled
|
||||
- dashboard upstream source degraded
|
||||
- sentinel stale
|
||||
- strategy or executor armed while critical input truth is stale
|
||||
|
||||
Each alert should include:
|
||||
- `alert_code`
|
||||
- `severity`
|
||||
- `service_scope`
|
||||
- clear reason text
|
||||
- relevant timestamps
|
||||
- threshold values used
|
||||
- enough details for replay and operator debugging
|
||||
|
||||
### 5. Add a quote-flow path check
|
||||
The recent incident was not merely “service disconnected.” It was “quote truth stopped.”
|
||||
|
||||
Add explicit checks for the active pair path:
|
||||
- last matching quote age
|
||||
- last raw quote persisted age if available
|
||||
- last normalized quote persisted age if available
|
||||
- mismatch between service-local quote timestamps and history-writer progress
|
||||
|
||||
This should allow the system to say whether the failure is:
|
||||
- upstream relay disconnected
|
||||
- ingest receiving traffic but not publishing
|
||||
- history path stalled after publish
|
||||
- dashboard path stale despite durable writes
|
||||
|
||||
### 6. Add bounded anomaly detection
|
||||
Do not build a free-text log learner. Use recent history and rolling baselines.
|
||||
|
||||
Initial anomaly detectors:
|
||||
- quote rate collapse over recent windows versus a rolling baseline
|
||||
- reconnect frequency spike over recent windows
|
||||
- durable-topic advancement collapse relative to recent baseline
|
||||
- raw-to-normalized or normalized-to-durable mismatch if both sides normally advance together
|
||||
|
||||
Implementation shape:
|
||||
- compute rolling counters in sentinel state or read recent durable counts from PostgreSQL
|
||||
- emit warning-level anomaly alerts or operator notices
|
||||
- keep deterministic alerts as the primary critical path
|
||||
|
||||
### 7. Add external alert delivery
|
||||
Add one notifier module and config for a webhook-oriented external sink.
|
||||
|
||||
Required behaviors:
|
||||
- deliver raised alerts
|
||||
- optionally deliver clear events
|
||||
- dedupe by alert identity
|
||||
- record delivery success or failure in service state
|
||||
- never block core alert state on delivery failure
|
||||
|
||||
Prefer a generic webhook shape that can feed:
|
||||
- Alertmanager receiver
|
||||
- Slack or Discord webhook
|
||||
- another downstream router
|
||||
|
||||
If a direct Alertmanager integration is convenient, keep the repo-owned payload explicit and narrow.
|
||||
|
||||
### 8. Add safe containment and remediation hooks
|
||||
Add only narrowly safe actions.
|
||||
|
||||
Allowed first-pass actions:
|
||||
- reconnect or refresh a service-local websocket client
|
||||
- pause or resume ingest-side or alert-side routines
|
||||
- disarm or pause strategy or executor when critical stale-data conditions are active
|
||||
|
||||
Containment policy for this turn:
|
||||
- if quote truth is critically stale, the system must at least surface a truthful safe control
|
||||
- if automatic containment is added, it must be non-fund-moving and clearly auditable
|
||||
|
||||
Do not add:
|
||||
- restarts of arbitrary deployments without explicit approval
|
||||
- trade retries
|
||||
- bridge, approval, or funding actions
|
||||
|
||||
### 9. Make the dashboard “angry”
|
||||
The dashboard should no longer use friendly reachability heuristics for serious failures.
|
||||
|
||||
Required UI changes:
|
||||
- global critical banner when critical alerts are active
|
||||
- service cards reflect derived severity, not just `health.ok`
|
||||
- stale services show exact age and why they are stale
|
||||
- funds page or relevant page clearly flags stale quotes
|
||||
- armed-state plus stale-data conditions render as critical
|
||||
|
||||
The specific failure mode `Freshness 30.8 h` plus `healthy` must become impossible.
|
||||
|
||||
### 10. Preserve durable truth
|
||||
Runtime health changes must remain durable and replayable.
|
||||
|
||||
Use existing paths where practical:
|
||||
- `ops.alert` stays the durable alert topic
|
||||
- `ops_alerts` remains the canonical alert store
|
||||
|
||||
If anomaly notices need separate storage, keep them closely aligned with existing alerting rather than creating a parallel history system without need.
|
||||
|
||||
## Concrete implementation order
|
||||
|
||||
### Phase 1. Define the health contract
|
||||
- enumerate service-local fields needed from each control API
|
||||
- define severity thresholds for each relevant service
|
||||
- define the first deterministic alert set
|
||||
- define the first anomaly set and its data sources
|
||||
|
||||
### Phase 2. Strengthen service state surfaces
|
||||
- update relevant `/state` and `/healthz` providers
|
||||
- expose websocket connection and freshness timestamps where missing
|
||||
- expose notifier and sentinel delivery state where needed
|
||||
|
||||
### Phase 3. Expand sentinel evaluation
|
||||
- add service polling
|
||||
- evaluate new deterministic conditions
|
||||
- reconcile raised and cleared alert state
|
||||
- keep current alert engine behavior passing
|
||||
|
||||
### Phase 4. Add quote-flow and pipeline checks
|
||||
- compare ingest-local state to durable progression
|
||||
- detect stalled or mismatched quote flow
|
||||
- emit new alerts with precise reasons
|
||||
|
||||
### Phase 5. Add anomaly detection
|
||||
- implement rolling or recent-window rate calculations
|
||||
- compare against baseline windows
|
||||
- emit warning-level anomaly alerts or notices
|
||||
|
||||
### Phase 6. Add external delivery
|
||||
- add notifier config and delivery module
|
||||
- send alert transitions to one external sink
|
||||
- expose delivery health and failures
|
||||
|
||||
### Phase 7. Make the dashboard severity truthful
|
||||
- replace soft healthy classification with derived severity
|
||||
- add loud banners and clearer service-card rendering
|
||||
- show freshness and cause directly
|
||||
|
||||
### Phase 8. Add safe containment
|
||||
- wire at least one truthful safe containment action
|
||||
- optionally auto-disarm or auto-pause on critical stale-data conditions if implemented cleanly
|
||||
- keep changes auditable and non-fund-moving
|
||||
|
||||
### Phase 9. Validate against the recent incident
|
||||
- reproduce or simulate a stranded ingest websocket condition
|
||||
- confirm durable alert, dashboard critical state, external notification, and containment behavior
|
||||
|
||||
## Test plan
|
||||
- unit tests for alert-engine raise and clear transitions for each new runtime-health alert
|
||||
- unit tests for derived service severity from freshness and disconnected state
|
||||
- unit tests for anomaly detectors on quote-rate collapse and reconnect spikes
|
||||
- unit tests for notifier dedupe and failure handling
|
||||
- integration tests for sentinel polling and alert emission using mocked service states
|
||||
- integration tests for dashboard bootstrap or reducer behavior under critical alerts and stale service snapshots
|
||||
- regression tests proving current price, inventory, funding, and executor submission alerts still work
|
||||
|
||||
No runtime-health bug fix is complete without a regression test covering the missed condition.
|
||||
|
||||
## Validation checklist against the proof
|
||||
- stale ingest is detected by policy, not by manual interpretation
|
||||
- dashboard shows warning or critical instead of healthy for stale or disconnected ingest
|
||||
- alert event is durable and replayable
|
||||
- external notification is emitted
|
||||
- at least one safe containment path exists
|
||||
- anomaly layer flags at least one non-predeclared abnormal pattern
|
||||
- live NEAR spendable truth remains unchanged
|
||||
|
||||
## Failure modes to plan for
|
||||
- service `/healthz` still says `ok` while runtime truth is stale
|
||||
- dashboard severity logic diverges from sentinel logic
|
||||
- alert delivery blocks or breaks alert generation
|
||||
- anomaly detection floods noise and hides real failures
|
||||
- containment silently changes arming state without durable evidence
|
||||
- service polling itself goes stale and no sentinel-stale alert fires
|
||||
|
|
@ -1,174 +0,0 @@
|
|||
# Implementation Proof: runtime health sentinel alert routing and anomaly detection
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-08
|
||||
|
||||
## Target outcome
|
||||
This turn proves that `unrip` can detect and loudly escalate broken runtime behavior before operators mistake stale or disconnected systems for healthy trading state.
|
||||
|
||||
The concrete target is the currently live NEAR Intents BTC/EURe system:
|
||||
- detect when quote flow stops or becomes stale
|
||||
- detect when websocket-backed services are disconnected or stranded
|
||||
- detect when durable pipeline components are reachable but no longer advancing truth
|
||||
- surface those failures as durable alerts and obvious dashboard state
|
||||
- deliver at least one external alert notification through a repo-controlled routing path
|
||||
- contain risk safely by exposing or applying only explicitly approved non-fund-moving remediation
|
||||
|
||||
## Why this is a meaningful architecture test
|
||||
The cluster incident on 2026-04-07 and 2026-04-08 showed a real failure mode:
|
||||
- `near-intents-ingest` stopped receiving and publishing quotes after `2026-04-07T09:06:38Z`
|
||||
- the service remained running in Kubernetes
|
||||
- the dashboard still rendered the service as effectively healthy despite `30h+` freshness age
|
||||
- no durable alert or external notification made the failure hard to miss
|
||||
|
||||
That is an architecture failure, not just an isolated websocket bug. If the system cannot detect that its truth pipeline has gone stale, it is not safe enough to broaden venue scope.
|
||||
|
||||
## Hypothesis
|
||||
`unrip` becomes more trustworthy if runtime health is treated as first-class product truth:
|
||||
- service-local state is not enough; cross-service health must be derived from the shared pipeline
|
||||
- stale or broken quote flow must become durable, replayable alert truth
|
||||
- dashboard health must be severity-driven, not merely reachability-driven
|
||||
- anomaly detection should complement hard invariants, not replace them
|
||||
- safe containment is more important than optimistic self-healing
|
||||
|
||||
The turn passes only if a quote-flow or service-health failure like the recent ingest outage would now produce obvious dashboard severity plus durable and external alert evidence within bounded time.
|
||||
|
||||
## Scope
|
||||
- [I020] Runtime health sentinel, alert routing, and anomaly detection: detect stale or broken service behavior across ingest, execution, persistence, and operator surfaces; surface critical failures loudly in the dashboard; emit durable alerts plus external notifications; and support tightly scoped safe remediation for non-fund-moving failures.
|
||||
- Extend the existing `ops-sentinel` into the runtime health authority for the current single-pair NEAR Intents system.
|
||||
- Cover at least these services and paths:
|
||||
- `near-intents-ingest`
|
||||
- `trade-executor`
|
||||
- `history-writer`
|
||||
- `operator-dashboard`
|
||||
- the quote-flow path from ingest to PostgreSQL-backed operator visibility
|
||||
- Add one repo-controlled external alert delivery path. Initial assumption: a webhook-oriented sink suitable for Alertmanager, Slack, Discord, or equivalent downstream routing.
|
||||
- Add bounded anomaly detection based on rolling baselines and pipeline gaps for the active pair and live services. This is support for operator review, not a replacement for deterministic safety alerts.
|
||||
|
||||
## Assumptions
|
||||
- The current control APIs remain the primary service-local truth surface for runtime state and safe controls.
|
||||
- External notification should be added through one simple sink first, not a broad escalation matrix in the repo itself.
|
||||
- Alertmanager or another downstream fan-out layer may exist later, but this turn only needs a truthful outbound integration point plus at least one working delivery path.
|
||||
- The first anomaly layer should use deterministic rolling windows or baseline comparisons over repo-owned metrics and history, not opaque model training over free-text logs.
|
||||
- Safe remediation in this turn means non-fund-moving actions only, such as reconnect, refresh, pause, resume, or explicit trading containment.
|
||||
|
||||
## Turn-shaping rules
|
||||
- This is a runtime-truth turn, not a generic observability platform buildout.
|
||||
- Do not add a broad metrics stack, tracing system, or large log-ML pipeline unless required by the proof.
|
||||
- Do not pretend anomaly detection is safety-complete. Deterministic invariants remain the primary alert path.
|
||||
- Do not let automatic remediation widen risk. If a condition is unsafe, containment or explicit alerting is preferred to speculative recovery.
|
||||
- Browser clients must still talk only to repo-owned backend services.
|
||||
|
||||
## Non-goals
|
||||
- No broad CoW Protocol implementation work in this turn beyond preserving the paused archive.
|
||||
- No generalized self-healing orchestration across arbitrary workloads.
|
||||
- No live-funds-moving automation such as bridge actions, funding, approvals, or trade submission retries.
|
||||
- No full-blown ML log classifier or LLM-operated runtime manager.
|
||||
- No replacement of Loki, Grafana, or external alert systems already in the cluster.
|
||||
|
||||
## Required runtime behavior
|
||||
|
||||
### Health truth
|
||||
The system must be able to derive runtime health from more than mere pod liveness.
|
||||
|
||||
At minimum it must detect and represent:
|
||||
- service reachable but stale
|
||||
- websocket disconnected
|
||||
- durable writer reachable but topic flow stalled
|
||||
- operator dashboard upstream source degraded
|
||||
- alerting subsystem stale or failing
|
||||
- strategy or executor armed while critical upstream truth is stale
|
||||
|
||||
### Alert truth
|
||||
The system must durably emit alert events for critical runtime failures.
|
||||
|
||||
At minimum it must raise and clear alerts for:
|
||||
- `near_intents_ingest_disconnected`
|
||||
- `near_intents_quotes_stale`
|
||||
- `near_intents_publish_stale`
|
||||
- `trade_executor_relay_disconnected`
|
||||
- `history_writer_stalled` or equivalent durable-write stall
|
||||
- `operator_dashboard_source_degraded`
|
||||
- `sentinel_stale`
|
||||
|
||||
The exact names may vary, but the semantic coverage must be equivalent and replayable from stored records.
|
||||
|
||||
### Operator surface
|
||||
Operators must be able to answer:
|
||||
- which service is degraded or critical
|
||||
- why it is degraded
|
||||
- how stale the underlying truth is
|
||||
- whether the issue is local-service, pipeline, or downstream-surface related
|
||||
- whether alert delivery succeeded
|
||||
- what safe control actions are available
|
||||
|
||||
The dashboard must no longer label a service healthy if its runtime truth is stale by policy.
|
||||
|
||||
### External notification
|
||||
At least one external notification path must receive a raised alert from repo-controlled code.
|
||||
|
||||
That path must:
|
||||
- dedupe by alert identity or equivalent key
|
||||
- include severity, scope, reason, and timestamps
|
||||
- support clear or recovery notification, or explicitly record why clear delivery is not supported
|
||||
|
||||
### Safe containment
|
||||
For at least one critical stale-data condition, the system must have a truthful containment path.
|
||||
|
||||
Initial target:
|
||||
- if ingest or quote truth is critically stale, operators can see a real safe action and the system can optionally force or recommend a non-fund-moving containment state such as pausing or disarming trade-driving components
|
||||
|
||||
### Bounded anomaly detection
|
||||
The system must add one anomaly layer beyond named invariants.
|
||||
|
||||
At minimum it must be able to flag:
|
||||
- quote rate collapse versus recent baseline
|
||||
- reconnect frequency spike versus recent baseline
|
||||
- topic-flow mismatch, such as raw or service-local activity without downstream durable progression
|
||||
|
||||
These anomaly signals may be warning-level and operator-facing rather than pager-level.
|
||||
|
||||
## Definition of done
|
||||
- The paused CoW turn is archived and the live turn files govern this runtime-health turn.
|
||||
- `ops-sentinel` or equivalent repo-owned runtime health authority evaluates service and pipeline health for the active live system.
|
||||
- Durable alert events represent the critical stale or disconnected conditions that the recent incident exposed.
|
||||
- The dashboard shows critical or stale health for those conditions rather than merely reporting large freshness ages under healthy badges.
|
||||
- At least one external alert route is wired and demonstrated from repo-owned code.
|
||||
- At least one safe containment action exists and is truthful.
|
||||
- At least one anomaly signal is implemented using recent-history comparisons or rolling baselines.
|
||||
- Tests cover alert raising and clearing, health classification, dashboard severity rendering, and alert delivery behavior.
|
||||
|
||||
For this turn to close with status `passed`, a reproduced or simulated quote-ingest failure must produce:
|
||||
- a durable raised alert
|
||||
- a clearly non-healthy dashboard state
|
||||
- an external notification
|
||||
- evidence of safe containment or explicit safe-control availability
|
||||
|
||||
## Validation evidence required
|
||||
- direct evidence that a stale or disconnected ingest condition is detected within a bounded threshold
|
||||
- direct evidence that the dashboard severity changes to warning or critical instead of healthy for that condition
|
||||
- direct evidence that the corresponding alert is stored durably
|
||||
- direct evidence that an external alert notification is emitted
|
||||
- automated test evidence for alert raise and clear transitions
|
||||
- automated test evidence for dashboard health classification from stale service state
|
||||
- automated test evidence for anomaly detection logic on at least one rolling-baseline case
|
||||
|
||||
## Failure conditions
|
||||
- The system still shows healthy or equivalent status while quote truth is stale by policy.
|
||||
- Alerts exist only in logs and are not durable or operator-visible.
|
||||
- External notification is absent or only manual.
|
||||
- Anomaly detection is described but not implemented in repo-owned code.
|
||||
- Safe containment would move funds or widen risk without explicit approval.
|
||||
- The turn produces only docs, dashboards, or thresholds without stronger runtime truth.
|
||||
|
||||
## Current real before this turn
|
||||
- The NEAR Intents BTC/EURe loop is live and already has durable history, strategy, execution, and dashboard surfaces.
|
||||
- `ops-sentinel` already emits some durable alerts for price, inventory, funding, and executor submission failures.
|
||||
- Grafana and Loki already exist in the cluster.
|
||||
- Service control APIs already expose state and safe controls.
|
||||
|
||||
## Deliberately not built by this proof
|
||||
- full multi-channel escalation policy management inside the repo
|
||||
- ML-first anomaly detection over raw free-text logs
|
||||
- broad auto-remediation beyond narrowly approved safe runtime actions
|
||||
- second-venue CoW execution work
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
import { formatAge, formatBoolean, formatEur, formatTimestamp, signedClass, truncateMiddle } from '../lib/format.js';
|
||||
import { SUBMISSION_COPY } from '../lib/submissionCopy.js';
|
||||
|
||||
function statusSubtitle(label, status, websocketState) {
|
||||
switch (label) {
|
||||
|
|
@ -9,8 +8,8 @@ function statusSubtitle(label, status, websocketState) {
|
|||
return formatTimestamp(status.market_observed_at);
|
||||
case 'Inventory Freshness':
|
||||
return formatTimestamp(status.inventory_observed_at);
|
||||
case SUBMISSION_COPY.statusTileLabel:
|
||||
return SUBMISSION_COPY.statusTileSubtitle;
|
||||
case 'Trading':
|
||||
return 'Successful submissions from durable history';
|
||||
default:
|
||||
return '';
|
||||
}
|
||||
|
|
@ -26,8 +25,8 @@ export default function StatusBar({ status, websocketState }) {
|
|||
['Alerts', `${status.active_alert_count || 0} ${status.highest_alert_severity ? `(${status.highest_alert_severity})` : ''}`.trim()],
|
||||
['Strategy Armed', formatBoolean(status.strategy_armed)],
|
||||
['Executor Armed', formatBoolean(status.executor_armed)],
|
||||
[SUBMISSION_COPY.statusTileLabel, `${status.recent_trade_count || 0} ${SUBMISSION_COPY.statusTileValueSuffix}`],
|
||||
[SUBMISSION_COPY.lastStatusTileLabel, formatTimestamp(status.last_successful_trade_at)],
|
||||
['Trading', `${status.recent_trade_count || 0} trades`],
|
||||
['Last Trade', formatTimestamp(status.last_successful_trade_at)],
|
||||
];
|
||||
|
||||
return (
|
||||
|
|
|
|||
|
|
@ -1,15 +0,0 @@
|
|||
export const SUBMISSION_COPY = {
|
||||
statusTileLabel: 'Submissions',
|
||||
statusTileSubtitle: 'Successful quote-response submissions from durable history',
|
||||
statusTileValueSuffix: 'submissions',
|
||||
lastStatusTileLabel: 'Last Submission',
|
||||
recentMetricLabel: 'Recent submissions',
|
||||
recentMetricValueSuffix: 'submissions',
|
||||
termsEyebrow: 'Submission activity',
|
||||
termsTitle: 'Recent submitted quote terms',
|
||||
termsEmpty: 'No submitted quote responses are available yet.',
|
||||
ledgerTitle: 'Submitted quote responses',
|
||||
ledgerSubtitle: 'PostgreSQL-backed pagination of executor submissions',
|
||||
ledgerEmpty: 'No submitted quote responses are stored yet.',
|
||||
ledgerCountLabel: (page, totalPages, total) => `Page ${page} of ${totalPages} - ${total} submitted quote responses`,
|
||||
};
|
||||
|
|
@ -5,7 +5,6 @@ import MetricCard from '../components/MetricCard.jsx';
|
|||
import Pill from '../components/Pill.jsx';
|
||||
import TableFrame from '../components/TableFrame.jsx';
|
||||
import { formatEur, formatTimestamp, stringifyJson, truncateMiddle } from '../lib/format.js';
|
||||
import { SUBMISSION_COPY } from '../lib/submissionCopy.js';
|
||||
|
||||
function buildInitialEstimateForm(balances, withdrawalDefaults) {
|
||||
const firstAssetId = balances?.[0]?.asset_id || '';
|
||||
|
|
@ -205,7 +204,7 @@ function QuotesTable({ items }) {
|
|||
}
|
||||
|
||||
function AssetChangeTable({ items }) {
|
||||
if (!items?.length) return <EmptyState>{SUBMISSION_COPY.termsEmpty}</EmptyState>;
|
||||
if (!items?.length) return <EmptyState>No successful trades are available yet.</EmptyState>;
|
||||
|
||||
return (
|
||||
<TableFrame>
|
||||
|
|
@ -214,8 +213,8 @@ function AssetChangeTable({ items }) {
|
|||
<tr>
|
||||
<th>Observed</th>
|
||||
<th>Quote</th>
|
||||
<th>Input</th>
|
||||
<th>Output</th>
|
||||
<th>Spend</th>
|
||||
<th>Receive</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
|
|
@ -234,7 +233,7 @@ function AssetChangeTable({ items }) {
|
|||
}
|
||||
|
||||
function TradesTable({ items }) {
|
||||
if (!items?.length) return <EmptyState>{SUBMISSION_COPY.ledgerEmpty}</EmptyState>;
|
||||
if (!items?.length) return <EmptyState>No successful trades are stored yet.</EmptyState>;
|
||||
|
||||
return (
|
||||
<TableFrame>
|
||||
|
|
@ -447,9 +446,9 @@ export default function FundsPage({
|
|||
value={formatEur(profitability.trading_contribution_eure)}
|
||||
/>
|
||||
<MetricCard
|
||||
label={SUBMISSION_COPY.recentMetricLabel}
|
||||
label="Recent trading"
|
||||
meta={formatTimestamp(profitability.last_successful_trade_at)}
|
||||
value={`${profitability.recent_trade_count || 0} ${SUBMISSION_COPY.recentMetricValueSuffix}`}
|
||||
value={`${profitability.recent_trade_count || 0} trades`}
|
||||
/>
|
||||
</div>
|
||||
<div className="panel-subtitle">
|
||||
|
|
@ -569,8 +568,8 @@ export default function FundsPage({
|
|||
<div className="panel">
|
||||
<div className="panel-head">
|
||||
<div>
|
||||
<div className="eyebrow">{SUBMISSION_COPY.termsEyebrow}</div>
|
||||
<h3>{SUBMISSION_COPY.termsTitle}</h3>
|
||||
<div className="eyebrow">Trade-driven changes</div>
|
||||
<h3>Recent asset deltas</h3>
|
||||
</div>
|
||||
</div>
|
||||
<AssetChangeTable items={funds.trade_asset_changes} />
|
||||
|
|
@ -581,13 +580,13 @@ export default function FundsPage({
|
|||
<div className="panel-head">
|
||||
<div>
|
||||
<div className="eyebrow">Durable ledger</div>
|
||||
<h3>{SUBMISSION_COPY.ledgerTitle}</h3>
|
||||
<h3>Successful trades</h3>
|
||||
</div>
|
||||
<div className="status-subtle">{SUBMISSION_COPY.ledgerSubtitle}</div>
|
||||
<div className="status-subtle">PostgreSQL-backed pagination</div>
|
||||
</div>
|
||||
<TradesTable items={trades.items} />
|
||||
<div className="pagination">
|
||||
<div className="status-subtle">{SUBMISSION_COPY.ledgerCountLabel(trades.page, trades.total_pages, trades.total)}</div>
|
||||
<div className="status-subtle">{`Page ${trades.page} of ${trades.total_pages} - ${trades.total} successful trades`}</div>
|
||||
<div className="button-row">
|
||||
<button
|
||||
className="button secondary"
|
||||
|
|
|
|||
|
|
@ -1,24 +0,0 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { SUBMISSION_COPY } from '../src/operator-dashboard/static/lib/submissionCopy.js';
|
||||
|
||||
test('submission copy does not present submissions as trades', () => {
|
||||
const text = [
|
||||
SUBMISSION_COPY.statusTileLabel,
|
||||
SUBMISSION_COPY.statusTileSubtitle,
|
||||
SUBMISSION_COPY.recentMetricLabel,
|
||||
SUBMISSION_COPY.termsEyebrow,
|
||||
SUBMISSION_COPY.termsTitle,
|
||||
SUBMISSION_COPY.termsEmpty,
|
||||
SUBMISSION_COPY.ledgerTitle,
|
||||
SUBMISSION_COPY.ledgerSubtitle,
|
||||
SUBMISSION_COPY.ledgerEmpty,
|
||||
SUBMISSION_COPY.ledgerCountLabel(1, 2, 3),
|
||||
].join(' ').toLowerCase();
|
||||
|
||||
assert.ok(text.includes('submission'));
|
||||
assert.ok(!text.includes('successful trades'));
|
||||
assert.ok(!text.includes('asset deltas'));
|
||||
assert.ok(!text.includes('trade-driven'));
|
||||
});
|
||||
Loading…
Add table
Reference in a new issue