unrip/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md

Implementation Turn: first non-mocked tradeable loop for one pair

Status: open Opened: 2026-04-01

Goal

Build the first full live vertical slice that can actually attempt a trade:

1. observe a real NEAR Intents quote
2. enrich it with live external reference pricing
3. synchronize spendable inventory already credited inside NEAR Intents
4. evaluate it in a real strategy service
5. gate it by inventory and arm state
6. submit it through a real Near Intents executor
7. persist the full chain in PostgreSQL

This turn is explicitly not a storage side-quest. Storage, control surfaces, analytics support, and treasury funding visibility are included because they are required to make the trade loop trustworthy.

Selected backlog items

• [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
• [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
• [I003] Strategy engine that consumes norm.swap_demand plus reference prices and emits auditable decisions.
• [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
• [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
• [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
• [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
• [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
• [B002] No reference price source exists, so the system cannot estimate edge.
• [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
• [B004] The current plan assumed external hot wallets were on the hot trade path instead of pre-funded internal inventory.

Architectural shape

Event backbone

Kafka or Redpanda remains the backbone between services.

Required topic set for this turn:

• raw.near_intents.quote
• norm.swap_demand
• ref.market_price
• state.intent_inventory
• ops.liquidity_action
• decision.trade_decision
• cmd.execute_trade
• exec.trade_result

Durable store

PostgreSQL is the first durable analytics and audit layer.

Why:

• enough write throughput for current scope
• strong queryability for replay, inspection, and debugging
• simple to operate
• better fit than inventing a warehouse right now

Service split

The first real loop should be seven services:

• near-intents-ingest
• market-reference-ingest
• inventory-sync
• liquidity-manager
• history-writer
• strategy-engine
• trade-executor

Service-by-service responsibilities

1. near-intents-ingest

Responsibilities:

• connect to the NEAR Intents quote stream
• filter or classify the active pair
• emit raw and normalized events
• expose runtime state

Inputs:

• NEAR Intents websocket
• pair filter config and runtime override

Outputs:

• raw.near_intents.quote
• norm.swap_demand

Control surface:

• GET /healthz
• GET /state
• GET /pair-filter
• PUT /pair-filter
• POST /pair-filter/reset

Important edge cases:

• websocket disconnect
• reconnect storm
• invalid JSON
• pair silent but connection healthy

2. market-reference-ingest

Responsibilities:

• subscribe to Kraken BTC/EUR pricing
• poll CoinGecko for fallback or cross-check pricing
• derive fair BTC/EURe price for both trade directions
• publish reference-price events
• expose latest source state and freshness

Inputs:

• Kraken stream
• CoinGecko HTTP API

Outputs:

• ref.market_price

Control surface:

• GET /healthz
• GET /state
• POST /refresh
• POST /pause
• POST /resume

Required state:

• latest Kraken price
• latest CoinGecko price
• derived fair rate
• freshness age
• source health flags

Important edge cases:

• Kraken disconnect
• CoinGecko timeout or rate limit
• both sources stale
• conflicting sources beyond tolerance

Required behavior:

• if Kraken is down but CoinGecko is fresh, degrade according to policy and record fallback usage
• if both are stale, mark the service unhealthy for decisioning
• the first implementation must make the EURe/EUR pricing assumption explicit:
  • either treat EURe as 1:1 with EUR and record that plainly
  • or use a separate sanity source and record the mapping logic

3. inventory-sync

Responsibilities:

• read current credited spendable inventory from NEAR Intents internal state
• distinguish credited balances from pending deposits and pending withdrawals
• publish current internal inventory state
• expose freshness and reconciliation state

Inputs:

• NEAR Intents inventory or verifier surfaces
• liquidity-manager state when relevant

Outputs:

• state.intent_inventory

Control surface:

• GET /healthz
• GET /state
• POST /refresh
• POST /pause
• POST /resume

Required state:

• spendable balances by asset
• pending inbound funding by asset
• pending outbound withdrawal by asset
• last sync time
• reconciliation status

Important edge cases:

• internal inventory surface unavailable
• external deposit seen but not yet credited internally
• credited balance lower than expected after funding
• stale inventory snapshot

Required behavior:

• only credited internal inventory counts as spendable
• pending treasury movements must remain non-spendable
• stale inventory state must be visible and actionable

4. liquidity-manager

Responsibilities:

• request and track deposit addresses or equivalent funding handles for treasury assets
• track treasury funding actions from external wallets into NEAR Intents
• track withdrawals and rebalance actions
• expose current funding pipeline state
• publish auditable liquidity action records

Inputs:

• treasury configuration
• external funding wallets
• NEAR Intents deposit or withdrawal surfaces

Outputs:

• ops.liquidity_action

Control surface:

• GET /healthz
• GET /state
• POST /refresh
• POST /pause
• POST /resume
• POST /freeze-withdrawals

Required state:

• active deposit addresses or funding handles
• recent funding attempts
• pending credits
• recent withdrawals
• rebalance state

Important edge cases:

• deposit address request failure
• external wallet funded but NEAR Intents credit delayed
• duplicate funding detection
• unsupported asset or chain mapping

Required behavior:

• treasury actions must remain visible and auditable
• funding must be decoupled from the per-trade hot path

5. history-writer

Responsibilities:

• consume the core topics
• write append-only rows into PostgreSQL
• preserve causality across quote, decision, and execution records
• expose lag and write state

Inputs:

• all core Kafka topics

Outputs:

• PostgreSQL rows

Control surface:

• GET /healthz
• GET /state
• POST /pause
• POST /resume
• POST /drain

Required stored record families:

• raw quotes
• normalized demand
• reference prices
• inventory snapshots
• liquidity actions
• trade decisions
• execute commands
• execution results

Important edge cases:

• PostgreSQL unavailable
• duplicate deliveries from Kafka
• offset commit mismatch after partial write

Required behavior:

• never claim success before the write is durable
• surface last committed offsets and last successful write time

6. strategy-engine

Responsibilities:

• join latest demand with latest price and inventory state
• compute implied quote rate
• compare it to fair rate
• apply freshness thresholds
• apply the initial 2% gross edge threshold
• apply max-notional and inventory checks
• emit auditable decision events
• emit cmd.execute_trade only when all gates pass and the system is armed

Inputs:

• norm.swap_demand
• ref.market_price
• state.intent_inventory

Outputs:

• decision.trade_decision
• cmd.execute_trade

Control surface:

• GET /healthz
• GET /state
• POST /arm
• POST /disarm
• POST /pause
• POST /resume
• PUT /threshold
• PUT /limits

Required decision state:

• arm state
• current threshold
• current notional cap
• latest decision
• latest rejected decision reason
• per-reason skip counters

Required decision record fields:

• decision_id
• quote_id
• pair
• direction
• implied_rate
• reference_rate
• gross_edge_pct
• price_freshness_ms
• inventory_snapshot
• decision
• decision_reason

Important edge cases:

• stale prices
• no price yet
• no inventory yet
• insufficient spendable inventory
• pending deposit exists but is not yet credited
• quote already expired
• repeated quote IDs
• one direction affordable and the other not
• fresh deploy starts armed by mistake

7. trade-executor

Responsibilities:

• consume cmd.execute_trade
• load the correct Near Intents signing authority
• perform the real Near Intents submission using pre-funded internal inventory
• preserve idempotency across retries and restarts
• emit exec.trade_result

Inputs:

• cmd.execute_trade
• signing secrets
• optional latest inventory state

Outputs:

• exec.trade_result

Control surface:

• GET /healthz
• GET /state
• POST /arm
• POST /disarm
• POST /pause
• POST /resume
• POST /drain

Required state:

• arm state
• last command seen
• last request sent
• last venue response
• in-flight command count
• completed command count
• error counters

Important edge cases:

• duplicate command delivery
• crash after submission but before result publish
• venue reject
• timeout with unknown outcome
• insufficient internal inventory detected late
• secret missing or invalid
• signer not authorized for the intended NEAR Intents account

Required behavior:

• never silently swallow a venue error
• always publish a result record for attempted commands
• make duplicate suppression durable
• never try to bridge or top up inventory during trade execution
• start disarmed by default on fresh deploy

Persistence design

Why PostgreSQL now

• current scale does not require a special time-series database
• rows are easier to inspect than a custom file format for this phase
• joins across decision, command, and result matter more right now than raw ingestion throughput
• treasury, inventory, and execution joins matter more than raw bus throughput at this stage

Minimum tables or equivalent record families

• raw_near_intents_quotes
• swap_demand_events
• market_price_events
• intent_inventory_snapshots
• liquidity_actions
• trade_decisions
• execute_trade_commands
• trade_execution_results

Query requirements

Must be able to answer:

• what was the latest fair price when this decision was made
• what spendable inventory existed when this decision was made
• what treasury funding actions were still pending
• why was this quote skipped
• what command was emitted for this quote
• what happened when the executor submitted it
• what credited internal inventory existed at the time

Control and stop semantics

Every service must support inspection.

State endpoints must be sufficient to answer:

• is it healthy
• is it connected
• what is it currently using as input state
• what was the last successful action
• why is it blocked, if blocked
• whether the state is authoritative or stale

Stopping must be explicit:

• pause means stop taking new work but keep process alive
• drain means finish in-flight work then stop cleanly
• disarm means remain live and observable but refuse side effects

This matters because the operator must be able to halt strategy or execution without destroying the whole pipeline.

Logging requirements

All services use structured JSON logs.

Stable top-level fields:

• level
• service
• component
• event
• namespace
• venue
• topic
• pair

Additional body fields when relevant:

• quote_id
• decision_id
• command_id
• execution_id
• inventory_id
• liquidity_action_id
• gross_edge_pct
• price_freshness_ms

Log when:

• source connection is lost or reestablished
• price source becomes stale
• inventory state becomes stale or recovers
• funding action is requested, seen, credited, delayed, failed, or frozen
• strategy rejects with a meaningful reason
• strategy arms or disarms
• executor arms or disarms
• command is submitted
• venue rejects or times out
• PostgreSQL disconnects or recovers
• control API changes state

Do not log every healthy message by default.

Failure and failover behavior

Reference pricing

• Kraken failure with fresh CoinGecko:
  • allowed only if fallback policy says yes
  • decision records must note fallback source use
• both sources stale:
  • strategy blocks all execution

Inventory state

• missing or stale inventory state:
  • strategy may still emit a non-actionable rejected decision
  • strategy must not emit cmd.execute_trade
• pending funding action:
  • remains non-spendable
  • must be visible in control state and durable records
• treasury action service down:
  • already funded trading may continue if inventory-sync is fresh
  • new funding or withdrawal operations must be blocked visibly

Persistence

• PostgreSQL unavailable:
  • history-writer unhealthy
  • system must surface that audit history is impaired
  • if the user wants strict mode, strategy or executor may be blocked until persistence returns

Execution

• timeout with unknown venue outcome:
  • emit explicit uncertain result
  • preserve idempotency state for recovery
• restart after partial submission:
  • executor must not blindly resubmit without checking prior state
• executor sees lower real inventory than strategy snapshot:
  • emit explicit failure result
  • do not attempt fallback treasury movement

Testing and validation plan

Unit tests

• implied-rate calculation for both directions
• explicit EURe/EUR pricing-basis test
• threshold checks
• stale-price blocking
• insufficient-inventory blocking
• pending-deposit-not-spendable blocking
• command emission only when armed
• idempotency transitions in executor

Integration tests

• pricing service emits canonical reference-price events
• inventory-sync emits credited vs pending inventory state correctly
• liquidity-manager records funding actions and status transitions
• strategy consumes demand, pricing, and inventory and emits decision plus command as expected
• history-writer persists linked records across topics
• executor publishes result records for success and failure paths

Runtime validation in cluster

• inspect each service /state
• verify live reference pricing updates
• verify live internal inventory state
• verify one treasury funding path from request or deposit tracking to credited inventory
• verify strategy and executor start disarmed on a fresh deploy
• observe a rejected decision due to block condition
• arm strategy and executor
• keep the first live max notional tiny, on the order of a few EURe, before any larger cap
• observe one command emission
• observe one real Near Intents execution attempt
• verify PostgreSQL contains the full linked chain

Deliberately rejected for this turn

• dashboards
• broad multi-pair abstractions
• ML training infrastructure
• broad backtest framework
• polished operator UI
• warehouse or lakehouse design

Expected deliverables

• market-reference-ingest
• inventory-sync
• liquidity-manager
• history-writer
• strategy-engine
• trade-executor
• PostgreSQL schema and migrations or equivalent setup
• updated Kafka topic creation and config
• shared control API pattern for all long-running services
• tests for strategy, inventory, persistence, and executor behavior
• docs for operations, arm or disarm flow, and inspection commands

Validation target

This turn is only complete when the deployed system can, through repo-controlled services alone, take one live active-pair quote, price it, verify credited internal inventory, decide on it, gate it, submit a real Near Intents execution attempt, and preserve the full record chain in PostgreSQL.