unrip/archive/implementation/20260404T075211Z-pre-credit-funding-visibility-and-operator-alerts-implementation.md

Implementation Turn: pre-credit funding visibility and operator alerts

Status: open Opened: 2026-04-02

Goal

Make the already-live BTC/EURe loop operationally understandable between external funding and spendable credit, and make critical stale or failed states queryable as durable alert records instead of transient logs.

Selected backlog items

• [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
• [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.

Design rule

Keep the already-proven execution and inventory truth path intact:

• verifier and bridge credit remain the only spendable truth
• pre-credit visibility is additive observability
• alerts are additive operability

Also keep the future operator architecture clean:

• no browser-direct Kafka or Redpanda dependency
• future dashboard reads PostgreSQL for history and service HTTP state for live controls and freshness
• this turn should therefore emit stable records and machine-friendly current-state summaries rather than log-only clues

Event backbone

Retain Kafka as the backbone. Add only the minimal new topics required:

• ops.funding_observation
• ops.alert

These topics are append-only evidence streams, not control planes.

Durable store

Extend PostgreSQL with new append-only families:

• funding_observations
• ops_alerts

If a current-state materialization is needed, derive it from append-only records or keep it as a clearly named snapshot table. Do not replace the append-only record.

Service changes

1. liquidity-manager

Extend the existing treasury owner instead of inventing a broad new funding stack.

New responsibilities:

• retain active funding handles by chain and asset
• poll configured chain observers for those handles
• emit ops.funding_observation
• correlate chain observations with bridge recent_deposits
• expose latest observations and credit-correlation state through /state

Expected state shape additions:

• funding_observations_by_handle
• latest_funding_observation_at
• uncredited_funding_total_by_asset
• credit_correlation
• observer_health
• observer_age_ms or equivalent freshness signal

Control additions:

• POST /refresh-funding-observations
• optional POST /pause-funding-observer
• optional POST /resume-funding-observer

Important implementation constraints:

• do not change withdrawal behavior
• do not reuse spendable inventory fields for pre-credit state
• keep BTC and EURe observation records in one shared schema

2. inventory-sync

Keep current spendable accounting intact.

Possible additions:

• read the latest funding observations
• expose a separate pre_credit_inbound or funding_visibility field in /state

Hard rule:

• spendable, pending_inbound, and strategy-facing credited truth must not become looser

3. history-writer

Consume and persist the new topics:

• ops.funding_observation
• ops.alert

Expose through /state:

• latest funding-observation write time
• latest alert write time
• counts or offsets for the new topics

Add query-friendly indexes for:

• tx_hash
• funding_handle
• alert_code
• status
• asset_id
• chain
• ingested_at

If this turn needs a current-state SQL materialization for future operator queries, keep it narrow and explicit, for example:

• latest funding observation per handle
• currently active alerts

Do not replace append-only records with mutable-only state.

4. ops-sentinel or equivalent alert evaluator

Add one small service only if needed to keep alert logic separate and testable.

Responsibilities:

• consume:
  • ref.market_price
  • state.intent_inventory
  • ops.liquidity_action
  • ops.funding_observation
  • exec.trade_result
• evaluate policy windows for stale and stuck conditions
• emit ops.alert raise/clear transitions
• expose current alert state

Preferred alert model:

• stable alert_code
• status: raised or cleared
• severity
• reason
• first_raised_at
• last_evaluated_at
• correlation IDs when available

State shape should be future-dashboard-friendly:

• active_alerts
• recent_transitions
• last_evaluated_at
• stale
• paused

Minimal alert set for this turn:

• reference_price_stale
• inventory_snapshot_stale
• funding_seen_unconfirmed
• funding_confirmed_credit_pending
• funding_stuck
• executor_submission_failed

Do not add Slack, email, or paging integrations in this turn unless required to prove the path. Durable alert records plus HTTP state are sufficient.

Chain observer plan

BTC

Must be the first-class proof path.

Implementation expectations:

• configurable observer endpoint
• look up the configured BTC deposit address
• detect:
  • mempool appearance when available
  • confirmation count
  • credited transition once bridge/verifier catches up

Assumption to keep explicit in code:

• a chain observer can disappear or lag independently of the bridge

Gnosis / EURe

Nice-to-have within the same schema, but BTC is the proof-critical path.

If included this turn:

• watch the configured deposit address for EURe token transfers
• represent observation state in the same event model

Record shapes

ops.funding_observation

Required fields:

• funding_observation_id
• account_id
• asset_id
• chain
• funding_handle
• source
• tx_hash
• status
• amount
• confirmations
• first_seen_at
• last_seen_at
• credited_at when known
• bridge_deposit_tx_hash when correlated

ops.alert

Required fields:

• alert_event_id
• alert_code
• status
• severity
• reason
• service_scope
• pair when relevant
• asset_id when relevant
• tx_hash when relevant
• raised_at
• cleared_at
• details

Control surface expectations

liquidity-manager

Must expose:

• active deposit handles
• latest pre-credit funding observations
• latest credit correlation
• whether the funding observer is healthy or paused
• explicit ages or timestamps so the future top bar can mark stale data without inferring it from logs

alert evaluator

Must expose:

• current active alerts
• latest cleared alerts
• per-alert evaluation timestamps
• pause state
• enough stable fields that a future dashboard backend can render alert lists and severity without custom log parsing

history-writer

Must expose the new topic offsets and write status for funding observations and alerts.

Tests

Required automated coverage:

• BTC funding observation remains non-spendable before credit
• alert transitions raise then clear on recovered stale state
• funding observation correlates to a later credited deposit without losing the original tx hash
• executor failure produces an alert event

If a meaningful automated test cannot be written for a subpath, stop and record why instead of hand-waving.

Validation plan

• Safe induced stale-price alert:
  • pause market-reference-ingest
  • wait past freshness window
  • observe reference_price_stale
  • resume and observe clear
• Safe induced stale-inventory alert:
  • pause inventory-sync
  • wait past freshness window
  • observe inventory_snapshot_stale
  • resume and observe clear
• Funding visibility proof:
  • use a real deposit address
  • observe pre-credit chain state before bridge credit where timing allows
  • later observe credit correlation
• Executor failure alert proof:
  • use a controlled non-destructive failure mode such as temporary relay endpoint override in a safe environment or a replayable failure fixture
  • verify executor_submission_failed

Future dashboard handoff

This turn should leave the next dashboard turn with these ready-made data sources:

• PostgreSQL history for funding observations and alerts
• service /state for current funding status, current alert status, balances, arming state, and freshness
• stable timestamps and reason fields that can drive:
  • a top status bar
  • a funds page
  • a system alert list

The next dashboard turn should not need to invent interpretation logic that belongs in this turn's services.

Out of scope on purpose

• No new trading strategy
• No historical backtest engine
• No broad observability stack
• No polished dashboard frontend
• No automated treasury refills

Still fake at turn open

• Pre-credit funding visibility is still missing from the live cluster.
• Alert state is still mostly implicit in service logs and manual inspection.
• There is no durable operator-facing record yet for "funds are on the way but not spendable."