unrip/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md

# Implementation Turn: first non-mocked tradeable loop for one pair

Status: open
Opened: 2026-04-01

## Goal
Build the first full live vertical slice that can actually attempt a trade:

1. observe a real NEAR Intents quote
2. enrich it with live external reference pricing
3. synchronize spendable inventory already credited inside NEAR Intents
4. evaluate it in a real strategy service
5. gate it by inventory and arm state
6. submit it through a real Near Intents executor
7. persist the full chain in PostgreSQL

This turn is explicitly not a storage side-quest. Storage, control surfaces, analytics support, and treasury funding visibility are included because they are required to make the trade loop trustworthy.

## Selected backlog items
- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions.
- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
- [B002] No reference price source exists, so the system cannot estimate edge.
- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
- [B004] The current plan assumed external hot wallets were on the hot trade path instead of pre-funded internal inventory.

## Architectural shape

### Event backbone
Kafka or Redpanda remains the backbone between services.

Required topic set for this turn:
- `raw.near_intents.quote`
- `norm.swap_demand`
- `ref.market_price`
- `state.intent_inventory`
- `ops.liquidity_action`
- `decision.trade_decision`
- `cmd.execute_trade`
- `exec.trade_result`

### Durable store
PostgreSQL is the first durable analytics and audit layer.

Why:
- enough write throughput for current scope
- strong queryability for replay, inspection, and debugging
- simple to operate
- better fit than inventing a warehouse right now

### Service split
The first real loop should be seven services:
- `near-intents-ingest`
- `market-reference-ingest`
- `inventory-sync`
- `liquidity-manager`
- `history-writer`
- `strategy-engine`
- `trade-executor`

## Service-by-service responsibilities

### 1. `near-intents-ingest`
Responsibilities:
- connect to the NEAR Intents quote stream
- filter or classify the active pair
- emit raw and normalized events
- expose runtime state

Inputs:
- NEAR Intents websocket
- pair filter config and runtime override

Outputs:
- `raw.near_intents.quote`
- `norm.swap_demand`

Control surface:
- `GET /healthz`
- `GET /state`
- `GET /pair-filter`
- `PUT /pair-filter`
- `POST /pair-filter/reset`

Important edge cases:
- websocket disconnect
- reconnect storm
- invalid JSON
- pair silent but connection healthy

### 2. `market-reference-ingest`
Responsibilities:
- subscribe to Kraken BTC/EUR pricing
- poll CoinGecko for fallback or cross-check pricing
- derive fair BTC/EURe price for both trade directions
- publish reference-price events
- expose latest source state and freshness

Inputs:
- Kraken stream
- CoinGecko HTTP API

Outputs:
- `ref.market_price`

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`

Required state:
- latest Kraken price
- latest CoinGecko price
- derived fair rate
- freshness age
- source health flags

Important edge cases:
- Kraken disconnect
- CoinGecko timeout or rate limit
- both sources stale
- conflicting sources beyond tolerance

Required behavior:
- if Kraken is down but CoinGecko is fresh, degrade according to policy and record fallback usage
- if both are stale, mark the service unhealthy for decisioning
- the first implementation must make the EURe/EUR pricing assumption explicit:
  - either treat EURe as 1:1 with EUR and record that plainly
  - or use a separate sanity source and record the mapping logic

### 3. `inventory-sync`
Responsibilities:
- read current credited spendable inventory from NEAR Intents internal state
- distinguish credited balances from pending deposits and pending withdrawals
- publish current internal inventory state
- expose freshness and reconciliation state

Inputs:
- NEAR Intents inventory or verifier surfaces
- liquidity-manager state when relevant

Outputs:
- `state.intent_inventory`

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`

Required state:
- spendable balances by asset
- pending inbound funding by asset
- pending outbound withdrawal by asset
- last sync time
- reconciliation status

Important edge cases:
- internal inventory surface unavailable
- external deposit seen but not yet credited internally
- credited balance lower than expected after funding
- stale inventory snapshot

Required behavior:
- only credited internal inventory counts as spendable
- pending treasury movements must remain non-spendable
- stale inventory state must be visible and actionable

### 4. `liquidity-manager`
Responsibilities:
- request and track deposit addresses or equivalent funding handles for treasury assets
- track treasury funding actions from external wallets into NEAR Intents
- track withdrawals and rebalance actions
- expose current funding pipeline state
- publish auditable liquidity action records

Inputs:
- treasury configuration
- external funding wallets
- NEAR Intents deposit or withdrawal surfaces

Outputs:
- `ops.liquidity_action`

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`
- `POST /freeze-withdrawals`

Required state:
- active deposit addresses or funding handles
- recent funding attempts
- pending credits
- recent withdrawals
- rebalance state

Important edge cases:
- deposit address request failure
- external wallet funded but NEAR Intents credit delayed
- duplicate funding detection
- unsupported asset or chain mapping

Required behavior:
- treasury actions must remain visible and auditable
- funding must be decoupled from the per-trade hot path

### 5. `history-writer`
Responsibilities:
- consume the core topics
- write append-only rows into PostgreSQL
- preserve causality across quote, decision, and execution records
- expose lag and write state

Inputs:
- all core Kafka topics

Outputs:
- PostgreSQL rows

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /pause`
- `POST /resume`
- `POST /drain`

Required stored record families:
- raw quotes
- normalized demand
- reference prices
- inventory snapshots
- liquidity actions
- trade decisions
- execute commands
- execution results

Important edge cases:
- PostgreSQL unavailable
- duplicate deliveries from Kafka
- offset commit mismatch after partial write

Required behavior:
- never claim success before the write is durable
- surface last committed offsets and last successful write time

### 6. `strategy-engine`
Responsibilities:
- join latest demand with latest price and inventory state
- compute implied quote rate
- compare it to fair rate
- apply freshness thresholds
- apply the initial 2% gross edge threshold
- apply max-notional and inventory checks
- emit auditable decision events
- emit `cmd.execute_trade` only when all gates pass and the system is armed

Inputs:
- `norm.swap_demand`
- `ref.market_price`
- `state.intent_inventory`

Outputs:
- `decision.trade_decision`
- `cmd.execute_trade`

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /arm`
- `POST /disarm`
- `POST /pause`
- `POST /resume`
- `PUT /threshold`
- `PUT /limits`

Required decision state:
- arm state
- current threshold
- current notional cap
- latest decision
- latest rejected decision reason
- per-reason skip counters

Required decision record fields:
- `decision_id`
- `quote_id`
- `pair`
- `direction`
- `implied_rate`
- `reference_rate`
- `gross_edge_pct`
- `price_freshness_ms`
- `inventory_snapshot`
- `decision`
- `decision_reason`

Important edge cases:
- stale prices
- no price yet
- no inventory yet
- insufficient spendable inventory
- pending deposit exists but is not yet credited
- quote already expired
- repeated quote IDs
- one direction affordable and the other not
- fresh deploy starts armed by mistake

### 7. `trade-executor`
Responsibilities:
- consume `cmd.execute_trade`
- load the correct Near Intents signing authority
- perform the real Near Intents submission using pre-funded internal inventory
- preserve idempotency across retries and restarts
- emit `exec.trade_result`

Inputs:
- `cmd.execute_trade`
- signing secrets
- optional latest inventory state

Outputs:
- `exec.trade_result`

Control surface:
- `GET /healthz`
- `GET /state`
- `POST /arm`
- `POST /disarm`
- `POST /pause`
- `POST /resume`
- `POST /drain`

Required state:
- arm state
- last command seen
- last request sent
- last venue response
- in-flight command count
- completed command count
- error counters

Important edge cases:
- duplicate command delivery
- crash after submission but before result publish
- venue reject
- timeout with unknown outcome
- insufficient internal inventory detected late
- secret missing or invalid
- signer not authorized for the intended NEAR Intents account

Required behavior:
- never silently swallow a venue error
- always publish a result record for attempted commands
- make duplicate suppression durable
- never try to bridge or top up inventory during trade execution
- start disarmed by default on fresh deploy

## Persistence design

### Why PostgreSQL now
- current scale does not require a special time-series database
- rows are easier to inspect than a custom file format for this phase
- joins across decision, command, and result matter more right now than raw ingestion throughput
- treasury, inventory, and execution joins matter more than raw bus throughput at this stage

### Minimum tables or equivalent record families
- `raw_near_intents_quotes`
- `swap_demand_events`
- `market_price_events`
- `intent_inventory_snapshots`
- `liquidity_actions`
- `trade_decisions`
- `execute_trade_commands`
- `trade_execution_results`

### Query requirements
Must be able to answer:
- what was the latest fair price when this decision was made
- what spendable inventory existed when this decision was made
- what treasury funding actions were still pending
- why was this quote skipped
- what command was emitted for this quote
- what happened when the executor submitted it
- what credited internal inventory existed at the time

## Control and stop semantics
Every service must support inspection.

State endpoints must be sufficient to answer:
- is it healthy
- is it connected
- what is it currently using as input state
- what was the last successful action
- why is it blocked, if blocked
- whether the state is authoritative or stale

Stopping must be explicit:
- `pause` means stop taking new work but keep process alive
- `drain` means finish in-flight work then stop cleanly
- `disarm` means remain live and observable but refuse side effects

This matters because the operator must be able to halt strategy or execution without destroying the whole pipeline.

## Logging requirements
All services use structured JSON logs.

Stable top-level fields:
- `level`
- `service`
- `component`
- `event`
- `namespace`
- `venue`
- `topic`
- `pair`

Additional body fields when relevant:
- `quote_id`
- `decision_id`
- `command_id`
- `execution_id`
- `inventory_id`
- `liquidity_action_id`
- `gross_edge_pct`
- `price_freshness_ms`

Log when:
- source connection is lost or reestablished
- price source becomes stale
- inventory state becomes stale or recovers
- funding action is requested, seen, credited, delayed, failed, or frozen
- strategy rejects with a meaningful reason
- strategy arms or disarms
- executor arms or disarms
- command is submitted
- venue rejects or times out
- PostgreSQL disconnects or recovers
- control API changes state

Do not log every healthy message by default.

## Failure and failover behavior

### Reference pricing
- Kraken failure with fresh CoinGecko:
  - allowed only if fallback policy says yes
  - decision records must note fallback source use
- both sources stale:
  - strategy blocks all execution

### Inventory state
- missing or stale inventory state:
  - strategy may still emit a non-actionable rejected decision
  - strategy must not emit `cmd.execute_trade`
- pending funding action:
  - remains non-spendable
  - must be visible in control state and durable records
- treasury action service down:
  - already funded trading may continue if inventory-sync is fresh
  - new funding or withdrawal operations must be blocked visibly

### Persistence
- PostgreSQL unavailable:
  - history-writer unhealthy
  - system must surface that audit history is impaired
  - if the user wants strict mode, strategy or executor may be blocked until persistence returns

### Execution
- timeout with unknown venue outcome:
  - emit explicit uncertain result
  - preserve idempotency state for recovery
- restart after partial submission:
  - executor must not blindly resubmit without checking prior state
- executor sees lower real inventory than strategy snapshot:
  - emit explicit failure result
  - do not attempt fallback treasury movement

## Testing and validation plan

### Unit tests
- implied-rate calculation for both directions
- explicit EURe/EUR pricing-basis test
- threshold checks
- stale-price blocking
- insufficient-inventory blocking
- pending-deposit-not-spendable blocking
- command emission only when armed
- idempotency transitions in executor

### Integration tests
- pricing service emits canonical reference-price events
- inventory-sync emits credited vs pending inventory state correctly
- liquidity-manager records funding actions and status transitions
- strategy consumes demand, pricing, and inventory and emits decision plus command as expected
- history-writer persists linked records across topics
- executor publishes result records for success and failure paths

### Runtime validation in cluster
- inspect each service `/state`
- verify live reference pricing updates
- verify live internal inventory state
- verify one treasury funding path from request or deposit tracking to credited inventory
- verify strategy and executor start disarmed on a fresh deploy
- observe a rejected decision due to block condition
- arm strategy and executor
- keep the first live max notional tiny, on the order of a few EURe, before any larger cap
- observe one command emission
- observe one real Near Intents execution attempt
- verify PostgreSQL contains the full linked chain

## Deliberately rejected for this turn
- dashboards
- broad multi-pair abstractions
- ML training infrastructure
- broad backtest framework
- polished operator UI
- warehouse or lakehouse design

## Expected deliverables
- `market-reference-ingest`
- `inventory-sync`
- `liquidity-manager`
- `history-writer`
- `strategy-engine`
- `trade-executor`
- PostgreSQL schema and migrations or equivalent setup
- updated Kafka topic creation and config
- shared control API pattern for all long-running services
- tests for strategy, inventory, persistence, and executor behavior
- docs for operations, arm or disarm flow, and inspection commands

## Validation target
This turn is only complete when the deployed system can, through repo-controlled services alone, take one live active-pair quote, price it, verify credited internal inventory, decide on it, gate it, submit a real Near Intents execution attempt, and preserve the full record chain in PostgreSQL.