unrip/archive/implementation/20260407T170647Z-operator-dashboard-foundation-funds-desk-and-operator-controls-implementation.md

# Implementation Turn: operator dashboard foundation, funds desk, and operator controls

Status: open
Opened: 2026-04-04

## Goal
Build a real operator dashboard service for the live BTC/EURe system so routine supervision no longer depends on ad hoc `curl`, `kubectl`, and SQL. The dashboard must read real durable and live state, make freshness obvious, and answer `are we making money` before it answers lower-level operator questions. It must also provide a narrow set of real operator controls without pretending that risky treasury actions are routine.

## Selected backlog items
- [I010] Operator dashboard foundation: dedicated dashboard pod plus backend API that reads PostgreSQL and service `/state`, with a global live status bar, arm/disarm controls, stale-state display, and `Funds` as the default home page.
- [I011] Funds desk: current balances, EUR-equivalent value, deposit-handle reveal with retrieval timestamps, emergency stop plus withdraw-all control path, deposit/withdraw history, trade-driven asset-change history, and flexible withdrawal forms by asset and EUR equivalent.
- [I014] Strategy and system operator pages: active strategy settings, skip reasons, alert state, service health, topic lag, persistence health, and cluster-operator stats.

## Design rule
The dashboard is a reader and operator surface over existing truths. It must not invent a second truth model.

- PostgreSQL remains the durable store for historical records and portfolio metrics.
- Existing services remain the owners of live state and control actions.
- The dashboard backend aggregates those sources and exposes one browser-facing API.
- The frontend must not talk to Kafka, Redpanda, PostgreSQL, or internal app services directly.
- Frontend polling is prohibited. Live updates must flow over WebSockets from the dashboard backend.

## Profitability-first rule
The default dashboard experience should answer these operator questions in order:
- what is the portfolio worth in EUR right now
- are we up or down versus the original deposits
- are we beating simple hold of the original deposits
- how much of the result appears to come from market move versus trading
- are trades actually happening right now

Operational telemetry should support that answer, not displace it.

## Deployment shape
Add one new service:
- `operator-dashboard`

Preferred shape:
- one Node service in the existing image
- serves static frontend assets plus JSON API plus WebSocket endpoint
- cluster-local by default, operator-accessed by port-forward or explicitly approved exposure later

Expected config additions:
- dashboard host or port
- PostgreSQL connection string
- Kafka brokers and a dedicated dashboard consumer group
- internal base URLs for:
  - `market-reference-ingest`
  - `inventory-sync`
  - `liquidity-manager`
  - `history-writer`
  - `ops-sentinel`
  - `strategy-engine`
  - `trade-executor`
  - optionally `near-intents-ingest`
- optional namespace or Kubernetes API settings if cluster state is included
- auth mode and any future operator-auth placeholders needed to keep the backend auth contract stable

## Backend architecture
The dashboard backend should have four cooperating parts:

1. REST API for bootstrap, history, and control actions
2. WebSocket hub for live push updates to the browser
3. PostgreSQL query layer for durable history and metrics
4. Kafka consumer layer for live event ingestion without polling

High-level flow:
- browser loads the page and calls dashboard REST endpoints for initial bootstrap data
- browser opens one backend WebSocket connection
- backend fans live events from Kafka into normalized WebSocket messages
- backend uses PostgreSQL for historical lists, pagination, and durable baseline data
- backend uses internal service REST calls for current control state and to execute allowlisted actions

The browser should never need its own polling timers.

## Auth skeleton
Add backend auth and authorization from the start, even if the first provider is permissive.

Required shape:
- a request-level auth middleware for REST
- a connection-level auth gate for WebSockets
- a structured auth context on every request or socket, for example:
  - `authenticated`
  - `subject`
  - `mode`
  - `roles` or equivalent

Initial implementation may be something like:
- `mode = stub`
- `authenticated = true`
- `subject = local-operator`

But route handlers and socket handlers must already depend on auth context rather than bypass auth entirely.

## Backend responsibilities

### 0. Authenticated API and socket surface
The backend should expose:
- a bootstrap or session route that tells the frontend whether it is authenticated
- authenticated JSON API routes
- an authenticated WebSocket channel for live dashboard updates

The goal is that future real auth can replace the provider, not the route architecture.

### 1. Aggregate durable history from PostgreSQL
The backend should query PostgreSQL directly for:
- latest portfolio metric from `portfolio_metrics_snapshots`
- deposit-time baseline and simple-hold benchmark inputs from credited deposit records plus reference-price history, or from durable precomputed portfolio metrics if those already exist
- recent deposits and withdrawals from the latest inventory snapshot
- recent funding observations from `funding_observations`
- active or recent alerts from `ops_alerts`
- recent raw or normalized quotes from:
  - `raw_near_intents_quotes`
  - or `swap_demand_events`, whichever is cleaner for the operator surface
- recent trade commands, results, and linked decision metadata from:
  - `execute_trade_commands`
  - `trade_execution_results`
  - `trade_decisions`
- latest market price and latest inventory snapshots as durable fallback truth

The backend should normalize these into dashboard-oriented response shapes rather than leaking raw SQL rows to the frontend.
If benchmark or attribution values are derived from incomplete data, the response shape should carry explicit caveats rather than silently presenting them as fully precise.

### 2. Aggregate live state from service control APIs
The backend should read `/state` and `/healthz` from the live services and normalize:
- freshness timestamps
- stale status
- pause state
- armed state
- last errors
- active alerts
- funding observer status
- writer offsets and database connectivity

The backend should make failures explicit per upstream service so one dead app does not blank the whole dashboard.
These reads should happen on demand for bootstrap and action-result flows, not through periodic polling loops.

### 3. Live event fan-out from Kafka to WebSockets
To satisfy the no-polling rule, the dashboard backend should consume Kafka directly and publish normalized WebSocket events.

Initial live topics should include at least:
- `raw.near_intents.quote` or `norm.swap_demand` for recent quote activity
- `ref.market_price` for price freshness and top-bar updates
- `state.intent_inventory` for balance freshness and valuation updates
- `ops.alert` for alert count and transition updates
- `exec.trade_result` for successful trade feed updates

The backend should maintain small in-memory current views such as:
- recent quote ring buffer limited to 10 items
- latest market price summary
- latest inventory summary
- active alert count or latest transitions

This keeps the browser live without polling while PostgreSQL remains the durable history source.

### 4. Optional read-only cluster state
For the system page, add a narrow read-only cluster view if practical:
- deployment readiness
- pod restart counts
- pod age
- PVC presence and mount health where useful

If this is included, use namespace-scoped read-only access only. Do not add broad cluster-admin behavior to satisfy a dashboard.
If live updates are needed here, prefer Kubernetes watch semantics or explicit refresh actions over polling loops.

## Frontend shape
Use the smallest frontend that can produce a real operator experience.

Preferred implementation:
- server-served static HTML, CSS, and JavaScript
- no dashboard-specific heavyweight frontend stack unless it materially reduces risk
- explicit operator refresh actions are acceptable
- polling is not acceptable

Expected browser behavior:
- one initial bootstrap fetch for page data
- one authenticated WebSocket connection for live updates
- no interval timers that refetch API routes

The UI should feel like a serious operator surface, not a placeholder admin page.

## Page plan

### Global status bar
The shared top bar should show:
- active pair
- latest EUR or EURe per BTC price
- market freshness and inventory freshness
- active alert count
- strategy armed state
- executor armed state
- total EUR portfolio value

The bar should color or label stale and degraded conditions explicitly.
The values in the bar should update from WebSocket events.

### Funds page
This is the default home page.

Sections to include:
- profitability summary at the top:
  - current total portfolio value in EUR
  - profit or loss versus deposit-time baseline
  - profit or loss versus simple hold of the original deposits
  - compact attribution between market move and trading contribution
  - last successful trade time and total successful trade count
  - explicit caveat text when fees or per-trade realized net profit are not fully tracked yet
- current balances by asset in nominal units
- EUR-equivalent valuation by asset and in total
- latest synced inventory timestamp and reconciliation status
- current funding handles and current funding observation state
- pre-credit funding visibility separated from spendable balances
- recent deposits and withdrawals
- recent trade-driven asset changes
- portfolio metric summary from the durable metric table

Control affordances on this page should prioritize safe actions already supported by the system:
- refresh funds or liquidity state
- pause or resume funding observer
- freeze or unfreeze withdrawals
- withdrawal estimate form by asset and amount

Actual live withdrawal submission is not part of this turn unless the user explicitly approves that risk during implementation.

### Activity section
Add a small operator activity surface, either as its own page or as a clearly named section.

It must include:
- recent incoming quotes: 10 most recent, live updated over WebSockets
- successful trades ledger: all successful trades, paginated from PostgreSQL

Keep this narrow:
- no bucket analytics
- no quote histograms
- no hindsight trade grading

Useful fields for recent quotes:
- observed time
- pair or asset direction
- request kind
- amount in and amount out when available

Useful fields for successful trades:
- time
- pair
- request kind
- submitted size
- result code
- linked decision edge when available

This activity surface should support the profitability view rather than compete with it. Recent trade count and last-trade recency should also feed the top-level profit summary.

### Strategy page
Sections to include:
- strategy armed and paused state
- executor armed, paused, and draining state
- threshold and notional settings
- recent decisions and skip reasons
- recent trade submission results
- alert state relevant to execution safety

If arm, disarm, pause, resume, or drain controls are shown here, the backend must classify them as risky or non-risky and require explicit confirmation for the risky ones.

### System page
Sections to include:
- service health summary across all runtime apps
- latest writer offsets and persistence health
- latest error surfaces by service
- current active alerts and recent transitions
- pod or deployment readiness and restart counts if cluster state is included
- any explicitly computed stale-state or degraded-state flags

This page should replace routine namespace health spot-checks for day-to-day operation, not full cluster administration.

## Control proxy plan
The backend should expose a small allowlisted control surface such as:
- `POST /api/control/:service/:action`

The backend should also expose WebSocket event shapes such as:
- `status_bar.updated`
- `quotes.recent`
- `alerts.updated`
- `inventory.updated`
- `trades.successful.appended`

The exact event names may differ, but they should be stable and explicit.

Rules:
- only known service/action pairs are allowed
- the backend maps them to existing internal control routes
- the response must include the updated live state when practical
- risky actions must require an explicit confirmation payload
- unknown or blocked actions must fail loudly

Safe controls to prioritize first:
- refresh actions
- pause or resume on non-fund-moving services
- funding observer pause or resume
- withdrawals freeze or unfreeze
- withdrawal estimate

Risky controls that may be shown but not exercised without explicit approval:
- strategy arm or disarm
- executor arm, disarm, or drain
- any control that can move funds

## Service changes expected in this turn
The dashboard should reuse existing service truth where possible, but a few additive changes are acceptable if they remove ambiguity:

### `history-writer`
May need extra read helpers or SQL query functions, but should remain the owner of portfolio-metric durability.

### `liquidity-manager`
May need clearer deposit-handle timestamps, withdrawal-freeze state detail, or withdrawal estimate response shaping for dashboard consumption.

### `strategy-engine` and `trade-executor`
May need cleaner state fields or explicit risk metadata for dashboard control rendering.

### `ops-sentinel`
Should remain the owner of current alert state; the dashboard should not re-implement alert logic in the browser.

### `operator-dashboard`
This new service becomes the owner of:
- browser authentication context
- dashboard REST contracts
- dashboard WebSocket contracts
- live quote ring buffer and other in-memory dashboard views fed from Kafka
- durable pagination queries for trade history and other operator lists

If a field is already available from an existing `/state`, do not duplicate it elsewhere just for convenience.

## Query and presentation requirements
The dashboard backend should present stable API responses that already answer operator questions such as:
- what do we hold right now?
- what is it worth in EUR right now?
- are we making money versus the deposit baseline?
- are we beating or lagging simple hold of the original deposits?
- how much of the current result appears to come from market move versus trading?
- what deposits are credited versus merely observed?
- what alerts are active right now and why?
- are strategy and executor armed?
- what happened in the most recent trades?
- what are the 10 most recent incoming quotes right now?
- what are all successful trades so far, across pages?
- which service is stale, unhealthy, or paused?

Avoid pushing interpretation complexity into the frontend when the backend can answer it once.

## Validation plan
- Deploy the dashboard service to k3s.
- Verify that the default page loads and shows live balances matching `inventory-sync` and PostgreSQL.
- Verify that the total EUR valuation matches the current reference price and current spendable inventory.
- Verify that the profitability summary matches a manual deposit-baseline and simple-hold calculation from durable history, including any caveat text where data is incomplete.
- Verify that the browser receives live updates over WebSockets with no polling timers or polling API loops.
- Verify that recent incoming quotes update live in the dashboard and remain capped to the 10 most recent items.
- Verify that the successful-trades list paginates through all successful trades from PostgreSQL.
- Verify that funding visibility shown on the dashboard matches `liquidity-manager` and `inventory-sync` current state.
- Safely induce a stale-price or stale-inventory condition and confirm the status bar and relevant page reflect it without manual SQL.
- Execute at least one safe control action through the dashboard backend and verify the underlying service state changes.
- Verify that a dead or unreachable upstream service degrades only its section rather than blanking the entire dashboard.
- Confirm that risky controls are visibly confirmation-gated and do not fire by accident.
- Verify that auth context exists on both REST and WebSocket paths even when the auth provider is permissive.

## Tests
Required automated coverage:
- REST auth middleware and WebSocket auth gating
- backend aggregation of PostgreSQL rows plus live service state into dashboard responses
- valuation logic for nominal and EUR-equivalent balances
- deposit-baseline, simple-hold, and attribution summary calculations, including caveat behavior when some inputs are incomplete
- active-alert and freshness normalization
- Kafka-to-WebSocket fan-out for recent quotes and other live dashboard updates
- recent-quote ring buffer behavior
- successful-trade pagination queries
- control routing allowlist and rejection behavior for unsupported actions
- confirmation gating for risky actions

If a dashboard rendering layer is non-trivial, add at least smoke coverage for the key pages or payload contracts that feed them.

## Kubernetes and runtime work
Expected deployment additions:
- `operator-dashboard` deployment and service
- config wiring for internal service URLs
- config wiring for Kafka access and dashboard consumer group
- optional RBAC if the dashboard reads Kubernetes readiness state

Keep RBAC narrow and read-only. No new privileged operator pod should be introduced just to make the dashboard convenient.

## Out of scope on purpose
- No full quote analytics workbench page beyond the narrow recent-quotes and successful-trades views
- No benchmark or hindsight analytics page
- No strategy redesign
- No browser-facing auth or multi-user access model
- No public ingress hardening work
- No one-click live withdrawal submit path without explicit approval

## Still fake at turn open
- There is no single operator dashboard service yet.
- Operators still depend on manual service HTTP calls and SQL for routine checks.
- No browser-safe backend aggregation layer exists yet.
- No backend auth layer exists yet for operator UI traffic.
- No dashboard WebSocket path exists yet.
- Cluster-level health is still surfaced through scripts and `kubectl`, not an in-product operator page.