philipp/unrip
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Archive first live trade loop and open funding visibility turn

Browse source 
Proof: Preserve the completed first live BTC/EURe trade loop and establish the next approved implementation proof around pre-credit funding visibility and operator alerts.
Assumptions: The live-trade loop is sufficiently proven by the recorded deposits, withdrawals, durable command/result chain, and successful mainnet quote responses; the next highest-value slice is operational visibility rather than new execution breadth.
Still fake: The newly opened funding-visibility and alert turn is planning only; no pre-credit watcher or durable alert evaluator is implemented yet.
This commit is contained in:
philipp 2026-04-03 01:07:02 +02:00
parent 16e7b79978
commit 54dc05a94c
20 changed files with 2234 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

162
AGENTS.md Normal file
View file

@ -0,0 +1,162 @@
# unrip agent rules
This repository is run with a data-first trading-system workflow.
## Core rule
Every change must directly serve the currently approved turn.
## What a turn is
A turn is the currently approved slice of work. There are two lanes:
- Implementation turn:
- defined by `PROOF.md` and `IMPLEMENTATION.md`
- the job is to implement what `IMPLEMENTATION.md` says, validate it against `PROOF.md`, and keep working until the scoped fake paths are removed or explicitly recorded as remaining blockers
- Research turn:
- defined by `research/ACTIVE.md`
- the job is to produce evidence against the active research charter, not product shape
`BACKLOG.md` is not the turn.
`ARCHIVE.md` is not the turn.
They are planning and history artifacts.
## Read order by task
Use the smallest context needed for the current task.
### For implementation work
Read only:
1. `THESIS.md`
2. `PROOF.md`
3. `IMPLEMENTATION.md`
Do not read `BACKLOG.md`, `ARCHIVE.md`, or `research/ACTIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, review history, or switch to research.
### For research work
Read only:
1. `THESIS.md`
2. `research/ACTIVE.md`
Do not read `BACKLOG.md`, `ARCHIVE.md`, `PROOF.md`, or `IMPLEMENTATION.md` unless the user explicitly asks to re-plan, compare lanes, or promote research into implementation.
### For planning, re-planning, or archiving
Read:
1. `THESIS.md`
2. the live turn files for the lane being changed
3. `BACKLOG.md`
4. `ARCHIVE.md`
## Hard constraints
- Do not invent or adopt a new roadmap on your own.
- Do not expand scope beyond the active implementation proof or research charter.
- No backlog generation instead of implementation.
- No scaffolding ahead of demonstrated need.
- Quote collection and analytics are first-class from day one. They are not a later add-on.
- Do not present scaffolding, dashboards, placeholders, or mock flows as product progress.
- State assumptions before coding when the environment, venue, chain, or source behavior is uncertain.
- Declare what is still fake in every commit.
- Do not read `BACKLOG.md` or `ARCHIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, or inspect history.
- If you discover adjacent work, add it to `BACKLOG.md` instead of absorbing it into the current turn.
- Changes that widen risk require explicit user approval:
- live funds
- secret creation or rotation
- permanent infrastructure spend
- long-running external jobs
- destructive data migrations
- The long-term thesis may be proposed, but `THESIS.md` must not be rewritten without explicit user approval.
## Iteration archive rule
When the user says to plan the next iteration, next turn, implementation turn, or proof sprint:
- first preserve the finished turn before drafting new planning docs
- do not rewrite the live turn files until the current turn has been archived
- use the existing repo workflow and archive locations:
- implementation turn snapshots go in `archive/implementation/`
- research turn snapshots go in `archive/research/`
- prefer the tracked scripts:
- `python3 scripts/workflow/close_turn.py ...`
- `python3 scripts/workflow/open_turn.py ...`
Planning or archiving is the one time it is correct to read `BACKLOG.md` and `ARCHIVE.md`.
## Planning inputs
- `iteration`, `implementation turn`, `proof sprint`, and `next turn` mean the same planning slice unless the user says otherwise.
- Use `BACKLOG.md` only while planning, re-planning, or archiving. Do not read it while implementing or validating the active turn unless the user explicitly asks.
- Select backlog items and bugs that belong together under one proof topic instead of mixing unrelated work into the same slice.
- When an item is pulled into the live turn, remove it from `BACKLOG.md` and record the planning event in `ARCHIVE.md`.
## Planning quality bar
- `IMPLEMENTATION.md` must be detailed enough that coding does not depend on rediscovering the plan mid-turn.
- `IMPLEMENTATION.md` should cover the end-to-end system touched by the proof: ingest, pricing, inventory, persistence, strategy, execution, control surfaces, logging, validation, tests, failure modes, and important edge cases.
- `PROOF.md` must be specific and falsifiable: scope, non-goals, definition of done, validation evidence, failure conditions, and clear statements about what is real versus fake.
- Both planning documents should think through the whole operator workflow and full system path, not isolated file edits.
## Turn lanes
- Implementation lane:
- Governed by `PROOF.md` and `IMPLEMENTATION.md`.
- Must make the shared live and historical data path more real unless the turn is explicitly marked as pure ops.
- “Doing the turn” means implementing `IMPLEMENTATION.md`, validating against `PROOF.md`, fixing what fails, and continuing until the definition of done is met or a hard blocker is reached.
- Research lane:
- Governed by `research/ACTIVE.md`.
- Must name the hypothesis, dataset, metric, assumptions, and falsification condition.
- Research output is evidence, not product shape.
## Commit rules
Every non-merge commit message body must include:
- `Proof: ...`
- `Assumptions: ...`
- `Still fake: ...`
Install the tracked hook before relying on this:
```bash
bash scripts/workflow/install_hooks.sh
```
## Post-implementation loop
- After implementation and validation, expect the user to test the slice and suggest small fixes.
- Treat those small fixes as part of closing the same turn unless the user explicitly changes scope.
## Bug-fix rule
- If a bug is found, fix it, inspect analogous or affected locations, and apply the needed follow-up fixes there too.
- No bug fix is done without a regression test.
- If a meaningful automated test cannot be added, stop and explain why instead of claiming the fix is complete.
## Real progress vs fake progress
Real progress means the repository can do more of the active proof with validated evidence from real systems. In this repo that means things like:
- ingesting real NEAR Intents flow
- storing durable inventory, pricing, decision, and execution records
- producing a real decision from live data
- making a real execution attempt through repo-controlled code
- proving blocked-path safety with explicit evidence
Fake progress includes:
- docs-only motion without stronger runtime truth
- speculative architecture not required by the active proof
- dashboards or control surfaces without the underlying real path
- placeholders or mocks presented as finished work
- invented data, unverifiable claims, or abstractions not yet required
## Safety rules
- Prefer the smallest real implementation that proves the active turn works.
- Keep secrets out of git, docs, and chat history.
- Use self-hosted or directly controlled infrastructure by default.
- If something is fake or incomplete, say so plainly.
## Review standard
Before claiming a turn is done, be able to state:
- what became more real
- what was validated against real data or systems
- what is still fake
- what was deliberately not built
## Working rule
Within an approved turn, continue implementing and validating until:
- `PROOF.md` is satisfied for the active scope, or
- a hard blocker requires user input.
Do not silently open the next turn yourself.
Do not stop at “the structure exists.”
Do not stop while the active turn still depends on hidden dummy paths.
If something remains fake at the end of the turn, name it plainly.

16
ARCHIVE.md Normal file
View file

@ -0,0 +1,16 @@
# Archive Index
This file records turn openings, closures, and archived snapshots.
Legacy note:
- Work completed before `2026-04-01` predates this workflow and is not retroactively indexed here.
## Implementation Turns
- 2026-04-02: `first-non-mocked-tradeable-loop-for-one-pair` closed with status `passed`. A live active-pair quote flowed through funding, inventory sync, strategy, and real mainnet quote responses with durable history.
## Research Turns
## Planning Events
- 2026-04-01: workflow files initialized for thesis, implementation proof, backlog, archive, and research lane.
- 2026-04-01: active implementation proof rewritten from durable-history scaffolding to the first executable trade loop for one pair.
- 2026-04-02: opened implementation turn `pre-credit-funding-visibility-and-operator-alerts` from backlog items O003, O004.

36
BACKLOG.md Normal file
View file

@ -0,0 +1,36 @@
# Backlog
This file is the candidate pool for future work. It is not the active plan.
Rules:
- Add ideas here when they do not belong in the current turn.
- Promotion from backlog to an active turn is a separate planning step.
- `scripts/workflow/add_backlog.py` can append new items with stable IDs.
## Implementation Candidates
- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions.
- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
- [I005] Import local EUR/BTC history on disk into research tables or files for replay and baseline checks.
- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
## Research Candidates
- [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair.
- [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold.
- [R003] Measure whether stale quotes correlate with worse execution quality or higher reject rates.
- [R004] Import and inspect the local historical EUR/BTC data to see how it can seed replay and backtests.
- [R005] Measure how long treasury funding takes from external transfer to credited internal inventory.
## Ops Candidates
- [O001] PostgreSQL backup and retention plan for analytics and audit history.
- [O002] Signing-secret and NEAR Intents account management for real execution credentials.
## Bugs
- [B001] The previous storage-only turn did not reach a tradeable loop and pulled the workflow toward scaffolding.
- [B002] No reference price source exists, so the system cannot estimate edge.
- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
- [B004] The prior plan assumed external hot wallets were on the trade hot path instead of pre-funded NEAR Intents inventory.

235
IMPLEMENTATION.md Normal file
View file

@ -0,0 +1,235 @@
# Implementation Turn: pre-credit funding visibility and operator alerts
Status: open
Opened: 2026-04-02
## Goal
Make the already-live BTC/EURe loop operationally understandable between external funding and spendable credit, and make critical stale or failed states queryable as durable alert records instead of transient logs.
## Selected backlog items
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
## Design rule
Keep the already-proven execution and inventory truth path intact:
- verifier and bridge credit remain the only spendable truth
- pre-credit visibility is additive observability
- alerts are additive operability
## Event backbone
Retain Kafka as the backbone. Add only the minimal new topics required:
- `ops.funding_observation`
- `ops.alert`
These topics are append-only evidence streams, not control planes.
## Durable store
Extend PostgreSQL with new append-only families:
- `funding_observations`
- `ops_alerts`
If a current-state materialization is needed, derive it from append-only records or keep it as a clearly named snapshot table. Do not replace the append-only record.
## Service changes
### 1. `liquidity-manager`
Extend the existing treasury owner instead of inventing a broad new funding stack.
New responsibilities:
- retain active funding handles by chain and asset
- poll configured chain observers for those handles
- emit `ops.funding_observation`
- correlate chain observations with bridge `recent_deposits`
- expose latest observations and credit-correlation state through `/state`
Expected state shape additions:
- `funding_observations_by_handle`
- `latest_funding_observation_at`
- `uncredited_funding_total_by_asset`
- `credit_correlation`
Control additions:
- `POST /refresh-funding-observations`
- optional `POST /pause-funding-observer`
- optional `POST /resume-funding-observer`
Important implementation constraints:
- do not change withdrawal behavior
- do not reuse spendable inventory fields for pre-credit state
- keep BTC and EURe observation records in one shared schema
### 2. `inventory-sync`
Keep current spendable accounting intact.
Possible additions:
- read the latest funding observations
- expose a separate `pre_credit_inbound` or `funding_visibility` field in `/state`
Hard rule:
- `spendable`, `pending_inbound`, and strategy-facing credited truth must not become looser
### 3. `history-writer`
Consume and persist the new topics:
- `ops.funding_observation`
- `ops.alert`
Expose through `/state`:
- latest funding-observation write time
- latest alert write time
- counts or offsets for the new topics
Add query-friendly indexes for:
- `tx_hash`
- `funding_handle`
- `alert_code`
- `ingested_at`
### 4. `ops-sentinel` or equivalent alert evaluator
Add one small service only if needed to keep alert logic separate and testable.
Responsibilities:
- consume:
- `ref.market_price`
- `state.intent_inventory`
- `ops.liquidity_action`
- `ops.funding_observation`
- `exec.trade_result`
- evaluate policy windows for stale and stuck conditions
- emit `ops.alert` raise/clear transitions
- expose current alert state
Preferred alert model:
- stable `alert_code`
- `status`: `raised` or `cleared`
- `severity`
- `reason`
- `first_raised_at`
- `last_evaluated_at`
- correlation IDs when available
Minimal alert set for this turn:
- `reference_price_stale`
- `inventory_snapshot_stale`
- `funding_seen_unconfirmed`
- `funding_confirmed_credit_pending`
- `funding_stuck`
- `executor_submission_failed`
Do not add Slack, email, or paging integrations in this turn unless required to prove the path. Durable alert records plus HTTP state are sufficient.
## Chain observer plan
### BTC
Must be the first-class proof path.
Implementation expectations:
- configurable observer endpoint
- look up the configured BTC deposit address
- detect:
- mempool appearance when available
- confirmation count
- credited transition once bridge/verifier catches up
Assumption to keep explicit in code:
- a chain observer can disappear or lag independently of the bridge
### Gnosis / EURe
Nice-to-have within the same schema, but BTC is the proof-critical path.
If included this turn:
- watch the configured deposit address for EURe token transfers
- represent observation state in the same event model
## Record shapes
### `ops.funding_observation`
Required fields:
- `funding_observation_id`
- `account_id`
- `asset_id`
- `chain`
- `funding_handle`
- `source`
- `tx_hash`
- `status`
- `amount`
- `confirmations`
- `first_seen_at`
- `last_seen_at`
- `credited_at` when known
- `bridge_deposit_tx_hash` when correlated
### `ops.alert`
Required fields:
- `alert_event_id`
- `alert_code`
- `status`
- `severity`
- `reason`
- `service_scope`
- `pair` when relevant
- `asset_id` when relevant
- `tx_hash` when relevant
- `raised_at`
- `cleared_at`
- `details`
## Control surface expectations
### `liquidity-manager`
Must expose:
- active deposit handles
- latest pre-credit funding observations
- latest credit correlation
- whether the funding observer is healthy or paused
### alert evaluator
Must expose:
- current active alerts
- latest cleared alerts
- per-alert evaluation timestamps
- pause state
### `history-writer`
Must expose the new topic offsets and write status for funding observations and alerts.
## Tests
Required automated coverage:
- BTC funding observation remains non-spendable before credit
- alert transitions raise then clear on recovered stale state
- funding observation correlates to a later credited deposit without losing the original tx hash
- executor failure produces an alert event
If a meaningful automated test cannot be written for a subpath, stop and record why instead of hand-waving.
## Validation plan
- Safe induced stale-price alert:
- pause `market-reference-ingest`
- wait past freshness window
- observe `reference_price_stale`
- resume and observe clear
- Safe induced stale-inventory alert:
- pause `inventory-sync`
- wait past freshness window
- observe `inventory_snapshot_stale`
- resume and observe clear
- Funding visibility proof:
- use a real deposit address
- observe pre-credit chain state before bridge credit where timing allows
- later observe credit correlation
- Executor failure alert proof:
- use a controlled non-destructive failure mode such as temporary relay endpoint override in a safe environment or a replayable failure fixture
- verify `executor_submission_failed`
## Out of scope on purpose
- No new trading strategy
- No historical backtest engine
- No broad observability stack
- No polished dashboard frontend
- No automated treasury refills
## Still fake at turn open
- Pre-credit funding visibility is still missing from the live cluster.
- Alert state is still mostly implicit in service logs and manual inspection.
- There is no durable operator-facing record yet for "funds are on the way but not spendable."

154
PROOF.md Normal file
View file

@ -0,0 +1,154 @@
# Implementation Proof: pre-credit funding visibility and operator alerts
Status: open
Opened: 2026-04-02
## Target outcome
The next turn is complete only when `unrip` can show operators the gap between "funds sent" and "funds spendable" with durable evidence, and can surface actionable alert state for the live loop without requiring log-diving or manual SQL every time something stalls.
This turn does not expand the trade hot path. It makes the existing live system more explainable and more operable.
## Hypothesis
`unrip` becomes materially safer to operate once it can:
1. observe configured funding handles before NEAR Intents credit
2. persist chain-level funding observations separately from spendable inventory
3. link pre-credit observations to later bridge and verifier credit where possible
4. emit durable alert state for stale prices, stale inventory, stuck funding, and failed execution submissions
5. expose that state through the same small control surfaces and PostgreSQL audit trail as the rest of the system
If pre-credit funding remains invisible, or alert state still lives only in transient logs, the live loop is still too opaque for routine funded operation.
## Scope
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
## Non-goals
- No new venue, pair, or strategy logic.
- No dashboard or polished UI.
- No automatic treasury actions or auto-refunding.
- No attempt to treat chain-level observations as spendable inventory.
- No change to the live execution arming model.
## Source-of-truth rule
Spendable inventory remains the existing truth:
- bridge and verifier credit determine spendable balances
- chain-level observations are visibility only
The new pre-credit path must never be allowed to make a direction tradable earlier than the verifier does.
## Required runtime behavior
### Funding visibility
- The system must know the currently active funding handles for BTC and EURe.
- For configured chains, it must watch those handles before NEAR Intents credit appears.
- It must distinguish at least these states where applicable:
- `SEEN_UNCONFIRMED`
- `SEEN_CONFIRMED`
- `CREDIT_PENDING`
- `CREDITED`
- `FAILED_OR_STUCK`
- BTC is the must-prove chain because that is where live funding latency was operationally visible.
- Gnosis support may share the same event model even if its confirmation behavior is simpler.
### Alerts
- Alerts must be durable records, not only log lines.
- At minimum the system must raise and clear alert state for:
- stale reference price
- stale inventory snapshot
- funding seen but not credited within policy
- execution submission failure
- Alert transitions must be inspectable through HTTP state and PostgreSQL.
## Service expectations
### `liquidity-manager`
Must become the owner of chain-level funding observations because it already owns deposit handles and treasury state.
It must:
- refresh and retain active funding handles
- ingest chain-level funding observations
- reconcile them against bridge deposit state
- publish durable funding-observation records
- expose current per-handle funding state
It must not:
- mark funds spendable
- trade on pre-credit observations
### `inventory-sync`
May surface pre-credit funding context, but only under a clearly separate non-spendable field.
It must not:
- merge pre-credit observations into `spendable`
### `history-writer`
Must persist the new record families:
- funding observations
- alert events
- optionally current alert snapshots if the implementation separates events from state
### Alert evaluator
This may be a new small service or a tightly scoped extension of an existing one, but it must:
- evaluate staleness and stuck conditions from durable inputs
- emit durable alert events
- expose current alert state and the latest reasons
No broad orchestration or dashboard service should be introduced just to satisfy this proof.
## Required durable storage
PostgreSQL must store at least:
- funding observations before bridge credit
- alert events or alert snapshots
- enough timestamps and IDs to correlate:
- funding handle
- chain tx hash
- later bridge tx hash or deposit record
- resulting verifier credit snapshot when available
Kafka remains the event backbone.
## Required control surface
At minimum operators must be able to inspect:
- active funding handles
- latest pre-credit observations by handle
- confirmation depth or equivalent chain state when available
- whether a funding action is still pending credit
- current active alerts and their reasons
If a new alert service exists, it must expose:
- `GET /healthz`
- `GET /state`
- `POST /pause`
- `POST /resume`
## Definition of done
- The live cluster still runs the previously proven funded trade loop unchanged for spendable truth.
- At least one real funding handle is watched at chain level before bridge credit.
- For at least one real deposit path, the system records a pre-credit observation before or during confirmation and later records the credited state separately.
- PostgreSQL contains durable records for the pre-credit funding path and alert path.
- Operators can inspect current funding observations and current alerts through control APIs.
- A stale price or stale inventory condition can be induced safely and becomes a durable alert.
- A funding delay or manually injected stuck condition can be represented as a durable alert with explicit reason fields.
- A failed execution submission path is represented as an alert without inventing fake venue traffic.
- Tests cover:
- pre-credit observations staying non-spendable
- alert raise and clear transitions
- correlation of funding observation to later credit when identifiers are available
## Failure conditions
- Funding observations exist only in logs and are not queryable later.
- Pre-credit observations leak into spendable inventory or strategy gating.
- Alerts cannot be queried as current state.
- The only proof of stuck funding is a human manually watching a block explorer.
- The implementation adds a dashboard shell without stronger runtime truth.
## Current real
- The first funded BTC/EURe live loop is already proven:
- real quote ingest
- real reference pricing
- real credited inventory
- real strategy decisions
- real `quote_response` submissions
- durable event chain in PostgreSQL
- Portfolio metrics are now durably computed and exposed, but alerting and pre-credit funding visibility are still incomplete.

50
THESIS.md Normal file
View file

@ -0,0 +1,50 @@
# unrip thesis
## Purpose
Build a data-first trading system whose first-class artifact is trustworthy market and execution history.
The bot is one consumer of that truth. Analytics and backtesting are not bolted on later; they are part of the product from the beginning.
## Product nucleus
The nucleus of the system is one shared truth pipeline:
1. observe live market and intent flow
2. persist raw and normalized events durably
3. replay the same history for analytics and backtests
4. score candidate actions from the same canonical data model
5. execute only behind explicit safety gates and full auditability
## Architectural invariants
- Live decisions and historical analysis must share the same canonical event model whenever practical.
- Raw events are kept alongside normalized and derived records.
- Every important decision should be reproducible from stored inputs and explicit assumptions.
- Execution must not outpace observability. If the system cannot explain what happened, it is not ready to trade.
- Quote collection and analytics are core product work, not support work.
## Near-term thesis
Near term, `unrip` should become a narrow but truthful trading-data and decision pipeline for one real pair on one venue.
That means:
- real upstream data
- durable storage beyond transient bus retention
- replayable history
- measurable candidate decisions
- no pretending that execution is safe before the data and analysis path is trustworthy
## Long-term thesis
Long term, the same system should support:
- automated trading on selected pairs
- analytics and backtesting from retained ground-truth data
- cross-chain and cross-asset execution routing
## Non-goals right now
- polished operator UI before the data loop is truthful
- broad multi-venue coverage before one core loop is real
- strategy claims without named assumptions and falsification criteria
- live trading with real funds before paper or tightly gated execution is trustworthy
## Approval boundaries
The agent may propose changes to the thesis, but the user must approve:
- changes to the core product definition
- changes that raise the risk class of the system
- changes that create lasting infra cost or operational burden

59
WORKFLOW.md Normal file
View file

@ -0,0 +1,59 @@
# Workflow
This repository uses a small tracked workflow layer instead of a large agent orchestration system.
## Files
- `THESIS.md`: stable product intent
- `PROOF.md`: active implementation proof
- `IMPLEMENTATION.md`: current implementation turn
- `research/ACTIVE.md`: active research charter
- `BACKLOG.md`: parked ideas and bugs
- `ARCHIVE.md`: turn history index
- `workflow/REVIEW_PROMPT.md`: adversarial review prompt
## Install the tracked git hook
```bash
bash scripts/workflow/install_hooks.sh
```
## Add a backlog item
```bash
python3 scripts/workflow/add_backlog.py --lane implementation --summary "Reference-price service for active pair inputs"
python3 scripts/workflow/add_backlog.py --lane research --summary "Test whether implied pair rate diverges from external reference prices after fees"
```
## Open a new implementation turn
```bash
python3 scripts/workflow/open_turn.py \
--lane implementation \
--title "first executable trade loop for one pair" \
--summary "Add reference pricing, strategy, durable audit history, and a real execution path for the active pair." \
--pick I001 \
--pick I002 \
--pick I003 \
--pick I004
```
Use `--commit` if you want the planning change committed automatically.
## Close the current turn and archive it
```bash
python3 scripts/workflow/close_turn.py \
--lane implementation \
--status passed \
--summary "A live active-pair quote flowed through pricing, decision, and a real execution attempt with durable audit records."
```
The script copies the live turn files into `archive/implementation/` or `archive/research/`, updates `ARCHIVE.md`, and can make the archive commit with `--commit`.
## Build a review bundle
```bash
bash scripts/workflow/review_diff.sh HEAD~1
```
That emits a Markdown bundle containing the diff plus the adversarial review prompt for a separate review-only agent session.

1
archive/implementation/.gitkeep Normal file
View file

@ -0,0 +1 @@

544
archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md Normal file
View file

@ -0,0 +1,544 @@
# Implementation Turn: first non-mocked tradeable loop for one pair
Status: open
Opened: 2026-04-01
## Goal
Build the first full live vertical slice that can actually attempt a trade:
1. observe a real NEAR Intents quote
2. enrich it with live external reference pricing
3. synchronize spendable inventory already credited inside NEAR Intents
4. evaluate it in a real strategy service
5. gate it by inventory and arm state
6. submit it through a real Near Intents executor
7. persist the full chain in PostgreSQL
This turn is explicitly not a storage side-quest. Storage, control surfaces, analytics support, and treasury funding visibility are included because they are required to make the trade loop trustworthy.
## Selected backlog items
- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions.
- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
- [B002] No reference price source exists, so the system cannot estimate edge.
- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
- [B004] The current plan assumed external hot wallets were on the hot trade path instead of pre-funded internal inventory.
## Architectural shape
### Event backbone
Kafka or Redpanda remains the backbone between services.
Required topic set for this turn:
- `raw.near_intents.quote`
- `norm.swap_demand`
- `ref.market_price`
- `state.intent_inventory`
- `ops.liquidity_action`
- `decision.trade_decision`
- `cmd.execute_trade`
- `exec.trade_result`
### Durable store
PostgreSQL is the first durable analytics and audit layer.
Why:
- enough write throughput for current scope
- strong queryability for replay, inspection, and debugging
- simple to operate
- better fit than inventing a warehouse right now
### Service split
The first real loop should be seven services:
- `near-intents-ingest`
- `market-reference-ingest`
- `inventory-sync`
- `liquidity-manager`
- `history-writer`
- `strategy-engine`
- `trade-executor`
## Service-by-service responsibilities
### 1. `near-intents-ingest`
Responsibilities:
- connect to the NEAR Intents quote stream
- filter or classify the active pair
- emit raw and normalized events
- expose runtime state
Inputs:
- NEAR Intents websocket
- pair filter config and runtime override
Outputs:
- `raw.near_intents.quote`
- `norm.swap_demand`
Control surface:
- `GET /healthz`
- `GET /state`
- `GET /pair-filter`
- `PUT /pair-filter`
- `POST /pair-filter/reset`
Important edge cases:
- websocket disconnect
- reconnect storm
- invalid JSON
- pair silent but connection healthy
### 2. `market-reference-ingest`
Responsibilities:
- subscribe to Kraken BTC/EUR pricing
- poll CoinGecko for fallback or cross-check pricing
- derive fair BTC/EURe price for both trade directions
- publish reference-price events
- expose latest source state and freshness
Inputs:
- Kraken stream
- CoinGecko HTTP API
Outputs:
- `ref.market_price`
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`
Required state:
- latest Kraken price
- latest CoinGecko price
- derived fair rate
- freshness age
- source health flags
Important edge cases:
- Kraken disconnect
- CoinGecko timeout or rate limit
- both sources stale
- conflicting sources beyond tolerance
Required behavior:
- if Kraken is down but CoinGecko is fresh, degrade according to policy and record fallback usage
- if both are stale, mark the service unhealthy for decisioning
- the first implementation must make the EURe/EUR pricing assumption explicit:
- either treat EURe as 1:1 with EUR and record that plainly
- or use a separate sanity source and record the mapping logic
### 3. `inventory-sync`
Responsibilities:
- read current credited spendable inventory from NEAR Intents internal state
- distinguish credited balances from pending deposits and pending withdrawals
- publish current internal inventory state
- expose freshness and reconciliation state
Inputs:
- NEAR Intents inventory or verifier surfaces
- liquidity-manager state when relevant
Outputs:
- `state.intent_inventory`
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`
Required state:
- spendable balances by asset
- pending inbound funding by asset
- pending outbound withdrawal by asset
- last sync time
- reconciliation status
Important edge cases:
- internal inventory surface unavailable
- external deposit seen but not yet credited internally
- credited balance lower than expected after funding
- stale inventory snapshot
Required behavior:
- only credited internal inventory counts as spendable
- pending treasury movements must remain non-spendable
- stale inventory state must be visible and actionable
### 4. `liquidity-manager`
Responsibilities:
- request and track deposit addresses or equivalent funding handles for treasury assets
- track treasury funding actions from external wallets into NEAR Intents
- track withdrawals and rebalance actions
- expose current funding pipeline state
- publish auditable liquidity action records
Inputs:
- treasury configuration
- external funding wallets
- NEAR Intents deposit or withdrawal surfaces
Outputs:
- `ops.liquidity_action`
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /refresh`
- `POST /pause`
- `POST /resume`
- `POST /freeze-withdrawals`
Required state:
- active deposit addresses or funding handles
- recent funding attempts
- pending credits
- recent withdrawals
- rebalance state
Important edge cases:
- deposit address request failure
- external wallet funded but NEAR Intents credit delayed
- duplicate funding detection
- unsupported asset or chain mapping
Required behavior:
- treasury actions must remain visible and auditable
- funding must be decoupled from the per-trade hot path
### 5. `history-writer`
Responsibilities:
- consume the core topics
- write append-only rows into PostgreSQL
- preserve causality across quote, decision, and execution records
- expose lag and write state
Inputs:
- all core Kafka topics
Outputs:
- PostgreSQL rows
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /pause`
- `POST /resume`
- `POST /drain`
Required stored record families:
- raw quotes
- normalized demand
- reference prices
- inventory snapshots
- liquidity actions
- trade decisions
- execute commands
- execution results
Important edge cases:
- PostgreSQL unavailable
- duplicate deliveries from Kafka
- offset commit mismatch after partial write
Required behavior:
- never claim success before the write is durable
- surface last committed offsets and last successful write time
### 6. `strategy-engine`
Responsibilities:
- join latest demand with latest price and inventory state
- compute implied quote rate
- compare it to fair rate
- apply freshness thresholds
- apply the initial 2% gross edge threshold
- apply max-notional and inventory checks
- emit auditable decision events
- emit `cmd.execute_trade` only when all gates pass and the system is armed
Inputs:
- `norm.swap_demand`
- `ref.market_price`
- `state.intent_inventory`
Outputs:
- `decision.trade_decision`
- `cmd.execute_trade`
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /arm`
- `POST /disarm`
- `POST /pause`
- `POST /resume`
- `PUT /threshold`
- `PUT /limits`
Required decision state:
- arm state
- current threshold
- current notional cap
- latest decision
- latest rejected decision reason
- per-reason skip counters
Required decision record fields:
- `decision_id`
- `quote_id`
- `pair`
- `direction`
- `implied_rate`
- `reference_rate`
- `gross_edge_pct`
- `price_freshness_ms`
- `inventory_snapshot`
- `decision`
- `decision_reason`
Important edge cases:
- stale prices
- no price yet
- no inventory yet
- insufficient spendable inventory
- pending deposit exists but is not yet credited
- quote already expired
- repeated quote IDs
- one direction affordable and the other not
- fresh deploy starts armed by mistake
### 7. `trade-executor`
Responsibilities:
- consume `cmd.execute_trade`
- load the correct Near Intents signing authority
- perform the real Near Intents submission using pre-funded internal inventory
- preserve idempotency across retries and restarts
- emit `exec.trade_result`
Inputs:
- `cmd.execute_trade`
- signing secrets
- optional latest inventory state
Outputs:
- `exec.trade_result`
Control surface:
- `GET /healthz`
- `GET /state`
- `POST /arm`
- `POST /disarm`
- `POST /pause`
- `POST /resume`
- `POST /drain`
Required state:
- arm state
- last command seen
- last request sent
- last venue response
- in-flight command count
- completed command count
- error counters
Important edge cases:
- duplicate command delivery
- crash after submission but before result publish
- venue reject
- timeout with unknown outcome
- insufficient internal inventory detected late
- secret missing or invalid
- signer not authorized for the intended NEAR Intents account
Required behavior:
- never silently swallow a venue error
- always publish a result record for attempted commands
- make duplicate suppression durable
- never try to bridge or top up inventory during trade execution
- start disarmed by default on fresh deploy
## Persistence design
### Why PostgreSQL now
- current scale does not require a special time-series database
- rows are easier to inspect than a custom file format for this phase
- joins across decision, command, and result matter more right now than raw ingestion throughput
- treasury, inventory, and execution joins matter more than raw bus throughput at this stage
### Minimum tables or equivalent record families
- `raw_near_intents_quotes`
- `swap_demand_events`
- `market_price_events`
- `intent_inventory_snapshots`
- `liquidity_actions`
- `trade_decisions`
- `execute_trade_commands`
- `trade_execution_results`
### Query requirements
Must be able to answer:
- what was the latest fair price when this decision was made
- what spendable inventory existed when this decision was made
- what treasury funding actions were still pending
- why was this quote skipped
- what command was emitted for this quote
- what happened when the executor submitted it
- what credited internal inventory existed at the time
## Control and stop semantics
Every service must support inspection.
State endpoints must be sufficient to answer:
- is it healthy
- is it connected
- what is it currently using as input state
- what was the last successful action
- why is it blocked, if blocked
- whether the state is authoritative or stale
Stopping must be explicit:
- `pause` means stop taking new work but keep process alive
- `drain` means finish in-flight work then stop cleanly
- `disarm` means remain live and observable but refuse side effects
This matters because the operator must be able to halt strategy or execution without destroying the whole pipeline.
## Logging requirements
All services use structured JSON logs.
Stable top-level fields:
- `level`
- `service`
- `component`
- `event`
- `namespace`
- `venue`
- `topic`
- `pair`
Additional body fields when relevant:
- `quote_id`
- `decision_id`
- `command_id`
- `execution_id`
- `inventory_id`
- `liquidity_action_id`
- `gross_edge_pct`
- `price_freshness_ms`
Log when:
- source connection is lost or reestablished
- price source becomes stale
- inventory state becomes stale or recovers
- funding action is requested, seen, credited, delayed, failed, or frozen
- strategy rejects with a meaningful reason
- strategy arms or disarms
- executor arms or disarms
- command is submitted
- venue rejects or times out
- PostgreSQL disconnects or recovers
- control API changes state
Do not log every healthy message by default.
## Failure and failover behavior
### Reference pricing
- Kraken failure with fresh CoinGecko:
- allowed only if fallback policy says yes
- decision records must note fallback source use
- both sources stale:
- strategy blocks all execution
### Inventory state
- missing or stale inventory state:
- strategy may still emit a non-actionable rejected decision
- strategy must not emit `cmd.execute_trade`
- pending funding action:
- remains non-spendable
- must be visible in control state and durable records
- treasury action service down:
- already funded trading may continue if inventory-sync is fresh
- new funding or withdrawal operations must be blocked visibly
### Persistence
- PostgreSQL unavailable:
- history-writer unhealthy
- system must surface that audit history is impaired
- if the user wants strict mode, strategy or executor may be blocked until persistence returns
### Execution
- timeout with unknown venue outcome:
- emit explicit uncertain result
- preserve idempotency state for recovery
- restart after partial submission:
- executor must not blindly resubmit without checking prior state
- executor sees lower real inventory than strategy snapshot:
- emit explicit failure result
- do not attempt fallback treasury movement
## Testing and validation plan
### Unit tests
- implied-rate calculation for both directions
- explicit EURe/EUR pricing-basis test
- threshold checks
- stale-price blocking
- insufficient-inventory blocking
- pending-deposit-not-spendable blocking
- command emission only when armed
- idempotency transitions in executor
### Integration tests
- pricing service emits canonical reference-price events
- inventory-sync emits credited vs pending inventory state correctly
- liquidity-manager records funding actions and status transitions
- strategy consumes demand, pricing, and inventory and emits decision plus command as expected
- history-writer persists linked records across topics
- executor publishes result records for success and failure paths
### Runtime validation in cluster
- inspect each service `/state`
- verify live reference pricing updates
- verify live internal inventory state
- verify one treasury funding path from request or deposit tracking to credited inventory
- verify strategy and executor start disarmed on a fresh deploy
- observe a rejected decision due to block condition
- arm strategy and executor
- keep the first live max notional tiny, on the order of a few EURe, before any larger cap
- observe one command emission
- observe one real Near Intents execution attempt
- verify PostgreSQL contains the full linked chain
## Deliberately rejected for this turn
- dashboards
- broad multi-pair abstractions
- ML training infrastructure
- broad backtest framework
- polished operator UI
- warehouse or lakehouse design
## Expected deliverables
- `market-reference-ingest`
- `inventory-sync`
- `liquidity-manager`
- `history-writer`
- `strategy-engine`
- `trade-executor`
- PostgreSQL schema and migrations or equivalent setup
- updated Kafka topic creation and config
- shared control API pattern for all long-running services
- tests for strategy, inventory, persistence, and executor behavior
- docs for operations, arm or disarm flow, and inspection commands
## Validation target
This turn is only complete when the deployed system can, through repo-controlled services alone, take one live active-pair quote, price it, verify credited internal inventory, decide on it, gate it, submit a real Near Intents execution attempt, and preserve the full record chain in PostgreSQL.

356
archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-proof.md Normal file
View file

@ -0,0 +1,356 @@
# Implementation Proof: first non-mocked tradeable loop for one pair
Status: open
Opened: 2026-04-01
## Target outcome
The active turn is not complete when the repo merely observes demand or emits placeholder commands. It is complete only when the deployed system can make a real Near Intents trade attempt for the active BTC/Gnosis EURe pair using pre-funded inventory already credited inside NEAR Intents, with the full chain stored durably for later analytics and backtesting.
## Hypothesis
`unrip` becomes materially more real once one live NEAR swap-demand event can move through this full chain without mocks:
1. real quote and demand ingestion
2. live external reference pricing
3. synchronized spendable inventory state from NEAR Intents
4. auditable strategy decision
5. inventory and safety gating
6. real Near Intents execution attempt using credited internal inventory
7. durable storage of the full event chain
If any of those links is still dummy, manual-only, or opaque after the fact, the proof is not achieved.
## Active pair and trading rule
- Active pair:
- `nep141:btc.omft.near`
- `nep141:gnosis-0x420ca0f9b9b604ce0fd9c18ef134c705e5fa3430.omft.near`
- Both directions must be implemented in the same decision and execution pipeline:
- BTC -> EURe
- EURe -> BTC
- Runtime spendable inventory inside NEAR Intents decides which direction can actually fire.
- External Gnosis and BTC wallets are treasury and funding inputs, not the per-trade spend path.
- Pending deposits or withdrawals must not be treated as spendable inventory.
- Initial decision rule:
- use Kraken plus CoinGecko reference prices
- make the EURe/EUR pricing assumption explicit in code and decision records
- require a gross edge of at least 2%
- require fresh reference pricing
- require fresh internal inventory state
- require sufficient spendable source-asset inventory already credited inside NEAR Intents
- keep strategy and executor disarmed by default on deploy
- start with a very small max notional for the first live attempt
## Required services
### `near-intents-ingest`
Responsibilities:
- connect to the live NEAR Intents quote feed
- maintain the active pair filter
- publish raw venue quotes
- publish normalized swap-demand events
- expose current ingest state
Must not:
- make trading decisions
- execute trades
### `market-reference-ingest`
Responsibilities:
- maintain current reference pricing for BTC/EUR and mapped BTC/EURe fair value
- use Kraken as the primary fast path
- use CoinGecko as fallback or cross-check
- publish reference-price events
- expose latest price state, freshness, and source health
Must not:
- emit trade commands
### `inventory-sync`
Responsibilities:
- read current spendable inventory from the NEAR Intents internal ledger or equivalent supported inventory surface
- distinguish credited inventory from pending deposits and withdrawals
- publish or serve current internal inventory state
- expose freshness, last successful sync time, and reconciliation status
Must not:
- initiate treasury movements
- make trading decisions
### `liquidity-manager`
Responsibilities:
- request or manage deposit addresses for supported treasury assets and chains
- track funding actions from external treasury wallets into NEAR Intents
- track pending deposits, credited deposits, withdrawals, and rebalance actions
- expose the current funding and treasury state
- publish or record liquidity actions so they are auditable
Must not:
- execute trading strategy
- spend inventory on the trade hot path
### `history-writer`
Responsibilities:
- consume the core Kafka topics
- write append-only records into PostgreSQL
- preserve enough IDs and timestamps to reconstruct causality
- expose writer lag, last committed offsets, and write health
Must not:
- serve as the message bus
- make trading decisions
### `strategy-engine`
Responsibilities:
- consume normalized demand plus fresh reference prices
- consume fresh spendable inventory state
- compute implied rate from the quote
- compare quote rate against external reference pricing
- apply the initial 2% gross edge threshold
- apply freshness, arming, and notional checks
- apply inventory-aware direction gating
- emit auditable decision events
- emit `cmd.execute_trade` only when a decision is actionable
Must not:
- hold private keys
- execute venue actions directly
### `trade-executor`
Responsibilities:
- consume `cmd.execute_trade`
- load the correct Near Intents signing authority at runtime
- perform the actual Near Intents submission against pre-funded internal inventory
- preserve idempotency and duplicate suppression
- publish `exec.trade_result`
- expose current arm state, recent attempts, and last venue error
Must not:
- invent trading logic
- bypass Kafka and manual safety gates
- perform ad hoc bridging or treasury funding on the trade hot path
## Required durable storage
PostgreSQL is the first durable analytics and audit store.
It must store at least:
- raw quotes
- normalized demand
- reference-price snapshots
- internal inventory snapshots
- liquidity and funding actions
- strategy decisions
- trade commands
- execution results
- treasury status and reconciliation metadata
Why PostgreSQL first:
- fast enough for the current streaming volume
- queryable for inspection and analytics
- simple to operate in the current cluster
- good fit for append-only audit plus current-state views
Kafka remains the streaming backbone. PostgreSQL is not the hot transport path.
## Required control surface
Every long-running service in this turn must expose a small HTTP control surface. The point is not a polished UI. The point is operability.
At minimum every service must expose:
- `GET /healthz`
- `GET /state`
Services with runtime controls must also expose the relevant action endpoints:
### `near-intents-ingest`
- inspect:
- pair filter
- connection state
- frames received
- published counts
- control:
- update pair filter
- disable or reset pair filter
### `market-reference-ingest`
- inspect:
- latest Kraken price
- latest CoinGecko price
- derived fair price
- freshness age
- source health
- control:
- pause or resume polling or streaming
- trigger ad hoc refresh
### `inventory-sync`
- inspect:
- latest spendable balances by asset
- pending deposits and withdrawals
- last sync time
- reconciliation status
- control:
- trigger sync
- pause or resume sync
### `liquidity-manager`
- inspect:
- active deposit addresses
- recent funding actions
- pending deposits
- credited deposits
- recent withdrawals
- control:
- request or rotate deposit address where supported
- refresh treasury status
- pause or resume funding trackers
- disable withdrawals or rebalance actions
### `history-writer`
- inspect:
- last persisted offsets by topic
- last write time
- error count
- database connectivity
- control:
- pause writes
- resume writes
- drain and stop cleanly
### `strategy-engine`
- inspect:
- arm state
- active threshold
- latest reference snapshot used
- latest inventory snapshot used
- latest decisions and reasons
- skipped counts by reason
- control:
- arm or disarm
- pause or resume decisions
- change threshold
- set notional cap
### `trade-executor`
- inspect:
- arm state
- last command received
- last venue response
- last error
- in-flight and completed command counts
- control:
- arm or disarm execution
- pause consumption
- drain and stop cleanly
Stopping a service must mean a graceful stop or drain, not “kill the pod and hope.”
## Logging and observability requirements
All services must emit structured JSON logs with stable fields.
Stable fields:
- `level`
- `service`
- `component`
- `event`
- `namespace`
- `venue`
- `topic`
- `pair`
High-cardinality IDs belong in the body, not labels:
- `quote_id`
- `command_id`
- `decision_id`
- `execution_id`
What must be logged:
- connection loss and recovery
- source stale state
- inventory-sync stale state
- treasury funding action requested, credited, failed, or stuck
- invalid messages
- decision accepted and decision rejected with reason
- arm and disarm actions
- execution submitted, rejected, failed, recovered, completed
- PostgreSQL disconnect and recovery
- control API failures
What should not be logged:
- per-message noise without state change
- full payload spam by default
## Required edge cases and failure handling
- stale reference prices must block decisions
- stale or missing internal inventory state must block execution
- insufficient credited inventory must block the affected direction only
- pending deposits must not be counted as spendable inventory
- deposit address created but never funded must not be mistaken for liquidity
- external transfer observed but not yet credited in NEAR Intents must stay non-spendable
- credited inventory drift between inventory-sync and executor must hard-fail the command and publish a result
- Kraken down but CoinGecko healthy should degrade, not crash, if policy allows fallback
- both reference sources stale must block decisions
- PostgreSQL down must surface a hard health failure and stop claiming the system is trade-ready
- liquidity-manager failure must block new funding operations but must not silently stop already funded trading
- duplicate `cmd.execute_trade` must not cause duplicate venue submission
- executor restart after a partial submission must preserve idempotency behavior
- Near Intents API or RPC failures must become explicit `trade_result` failure records
- pair inactivity must not be mistaken for pipeline breakage
## Definition of done
- The deployed cluster is running all required services for this loop.
- Live Kraken and CoinGecko reference prices are flowing and inspectable.
- Spendable inventory inside NEAR Intents is flowing and inspectable.
- At least one treasury funding path is documented and verified:
- deposit address or supported funding path created
- funding action observed
- credited inventory visible in internal state
- PostgreSQL contains the full event chain for at least one real quote path.
- Strategy decisions are non-dummy and include explicit reason fields with edge, freshness, and inventory context.
- Both trade directions are implemented in the same decision and execution path.
- At least one direction can be armed based on available credited inventory.
- A real Near Intents execution attempt can be triggered through the repo-controlled path.
- The resulting venue response is captured durably and inspectably.
- Each service exposes the required health and state endpoints, and controlled pause or arm semantics where appropriate.
- Fresh deploys start disarmed by default and require explicit operator arming before side effects.
- Validation includes not only happy path but blocked-path evidence:
- stale reference blocked
- insufficient inventory blocked
- pending-deposit-not-spendable blocked
- disarmed executor blocked
## Current real
- Real NEAR Intents quote data is flowing into the cluster.
- The pair filter can be configured and changed at runtime.
- Raw and normalized events already exist in Redpanda topics.
- The app is deployable and observable in Kubernetes.
- Kafka already provides the event backbone.
## Current fake or incomplete
- Reference pricing does not exist yet.
- Spendable internal inventory sync does not exist yet.
- Treasury funding or deposit management does not exist yet.
- Strategy decisions are still dummy placeholders.
- Execution is still dummy.
- The current executor is not a real Near Intents adapter.
- PostgreSQL is not yet part of the loop.
- Most services do not yet have their own control surfaces.
- Real signing and treasury secret handling is not yet wired.
## Failure conditions
This proof fails if any of the following is true:
- a trade can only happen via manual shell steps or out-of-band operator intervention
- the system depends on ad hoc bridging or external-wallet spending on the trade hot path
- decision logic is still embedded in a dummy or placeholder service
- executor logic is still simulated
- a service cannot be inspected or paused cleanly at runtime
- the system cannot explain why a quote did or did not become a trade
- the stored history cannot reconstruct quote, pricing, decision, command, and result
- both directions are not implemented
- the system attempts to execute without credited internal source inventory
## Expected validation evidence
- live `ref.market_price` events and current state from the pricing control API
- live internal inventory state from the inventory-sync control API
- PostgreSQL rows linking quote, reference, inventory snapshot, decision, command, and execution result
- a blocked decision caused by stale price or insufficient inventory
- a funding action that becomes credited inventory
- a successful command emission from the strategy service with a recorded edge above 2%
- one real Near Intents execution attempt with stored request and response metadata

1
archive/research/.gitkeep Normal file
View file

@ -0,0 +1 @@

14
research/ACTIVE.md Normal file
View file

@ -0,0 +1,14 @@
# Research Turn
Status: idle
No approved research turn is active yet.
When opening one, capture:
- charter
- hypothesis
- dataset or source of truth
- metrics
- assumptions
- falsification condition
- expected artifact paths under `research/experiments/`

9
research/QUEUE.md Normal file
View file

@ -0,0 +1,9 @@
# Research Queue
This queue holds strategy and analysis questions that are not yet an active research turn.
## Candidate charters
- Determine which Kraken symbols and CoinGecko asset IDs give a trustworthy pricing basis for the active pair.
- Measure how often the active pair's implied rate diverges from external reference prices after a simple fee model.
- Test whether stale quotes correlate with worse downstream execution quality or higher reject rates.
- Import the local historical EUR/BTC data and decide how it should seed replay and backtesting.

43
scripts/workflow/add_backlog.py Executable file
View file

@ -0,0 +1,43 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
from common import BACKLOG_PATH, BACKLOG_SECTION, insert_into_section, load_text, next_backlog_id, save_text
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Append an item to BACKLOG.md.")
parser.add_argument(
"--lane",
required=True,
choices=("implementation", "research", "ops", "bug"),
help="Backlog section to append to.",
)
parser.add_argument("--summary", required=True, help="One-line backlog summary.")
parser.add_argument(
"--priority",
default="soon",
choices=("now", "soon", "later"),
help="Coarse urgency label.",
)
parser.add_argument(
"--tags",
default="",
help="Comma-separated tags stored inline for grepability.",
)
return parser.parse_args()
def main() -> None:
args = parse_args()
item_id = next_backlog_id(args.lane)
tags = f" tags={args.tags}" if args.tags else ""
entry = f"- [{item_id}] ({args.priority}) {args.summary}{tags}"
updated = insert_into_section(load_text(BACKLOG_PATH), BACKLOG_SECTION[args.lane], entry)
save_text(BACKLOG_PATH, updated)
print(item_id)
if __name__ == "__main__":
main()

150
scripts/workflow/close_turn.py Executable file
View file

@ -0,0 +1,150 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import re
import shutil
from pathlib import Path
from common import (
ARCHIVE_PATH,
IMPLEMENTATION_ARCHIVE_DIR,
IMPLEMENTATION_PATH,
PROOF_PATH,
RESEARCH_ACTIVE_PATH,
RESEARCH_ARCHIVE_DIR,
append_archive_line,
git_commit,
load_text,
save_text,
slugify,
timestamp_slug,
today_iso,
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Archive and close the current implementation or research turn.")
parser.add_argument("--lane", required=True, choices=("implementation", "research"))
parser.add_argument(
"--status",
required=True,
choices=("passed", "failed", "paused", "abandoned"),
help="Outcome of the turn.",
)
parser.add_argument("--summary", required=True, help="One-line closure summary.")
parser.add_argument(
"--commit",
action="store_true",
help="Commit the archive change automatically.",
)
return parser.parse_args()
def title_from_content(content: str, prefix: str) -> str:
match = re.search(rf"^# {re.escape(prefix)}: (.+)$", content, flags=re.MULTILINE)
if not match:
raise SystemExit(f"could not find title for {prefix.lower()}")
return match.group(1).strip()
def close_implementation_turn(status: str, summary: str) -> tuple[str, list[Path]]:
proof_content = load_text(PROOF_PATH)
implementation_content = load_text(IMPLEMENTATION_PATH)
title = title_from_content(proof_content, "Implementation Proof")
slug = slugify(title)
stamp = timestamp_slug()
IMPLEMENTATION_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
proof_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-proof.md"
implementation_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-implementation.md"
shutil.copyfile(PROOF_PATH, proof_archive)
shutil.copyfile(IMPLEMENTATION_PATH, implementation_archive)
save_text(
PROOF_PATH,
"""# Implementation Proof
Status: idle
No approved implementation proof is active yet.
""",
)
save_text(
IMPLEMENTATION_PATH,
"""# Implementation Turn
Status: idle
No approved implementation turn is active yet.
""",
)
append_archive_line(
"implementation",
f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}",
)
return title, [proof_archive, implementation_archive]
def close_research_turn(status: str, summary: str) -> tuple[str, list[Path]]:
research_content = load_text(RESEARCH_ACTIVE_PATH)
title = title_from_content(research_content, "Research Turn")
slug = slugify(title)
stamp = timestamp_slug()
RESEARCH_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
research_archive = RESEARCH_ARCHIVE_DIR / f"{stamp}-{slug}.md"
shutil.copyfile(RESEARCH_ACTIVE_PATH, research_archive)
save_text(
RESEARCH_ACTIVE_PATH,
"""# Research Turn
Status: idle
No approved research turn is active yet.
""",
)
append_archive_line(
"research",
f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}",
)
return title, [research_archive]
def main() -> None:
args = parse_args()
if args.lane == "implementation" and "Status: idle" in load_text(PROOF_PATH):
raise SystemExit("no active implementation turn to close")
if args.lane == "research" and "Status: idle" in load_text(RESEARCH_ACTIVE_PATH):
raise SystemExit("no active research turn to close")
if args.lane == "implementation":
title, archived_paths = close_implementation_turn(args.status, args.summary)
else:
title, archived_paths = close_research_turn(args.status, args.summary)
if args.commit:
paths = [ARCHIVE_PATH]
if args.lane == "implementation":
paths.extend([PROOF_PATH, IMPLEMENTATION_PATH])
else:
paths.append(RESEARCH_ACTIVE_PATH)
paths.extend(archived_paths)
git_commit(
f"""Archive {args.lane} turn: {title}
Proof: Preserve the completed {args.lane} turn and record its outcome in the tracked archive.
Assumptions: The archived files capture the relevant planning state for the completed turn.
Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.""",
paths=paths,
)
for archived_path in archived_paths:
print(archived_path.relative_to(RESEARCH_ACTIVE_PATH.parent.parent))
if __name__ == "__main__":
main()

154
scripts/workflow/common.py Executable file
View file

@ -0,0 +1,154 @@
#!/usr/bin/env python3
from __future__ import annotations
import re
import subprocess
from datetime import datetime, timezone
from pathlib import Path
ROOT_DIR = Path(__file__).resolve().parents[2]
BACKLOG_PATH = ROOT_DIR / "BACKLOG.md"
ARCHIVE_PATH = ROOT_DIR / "ARCHIVE.md"
PROOF_PATH = ROOT_DIR / "PROOF.md"
IMPLEMENTATION_PATH = ROOT_DIR / "IMPLEMENTATION.md"
RESEARCH_ACTIVE_PATH = ROOT_DIR / "research/ACTIVE.md"
IMPLEMENTATION_ARCHIVE_DIR = ROOT_DIR / "archive/implementation"
RESEARCH_ARCHIVE_DIR = ROOT_DIR / "archive/research"
LANE_PREFIX = {
"implementation": "I",
"research": "R",
"ops": "O",
"bug": "B",
}
BACKLOG_SECTION = {
"implementation": "## Implementation Candidates",
"research": "## Research Candidates",
"ops": "## Ops Candidates",
"bug": "## Bugs",
}
ARCHIVE_SECTION = {
"implementation": "## Implementation Turns",
"research": "## Research Turns",
"planning": "## Planning Events",
}
def now_utc() -> datetime:
return datetime.now(timezone.utc)
def today_iso() -> str:
return now_utc().date().isoformat()
def timestamp_slug() -> str:
return now_utc().strftime("%Y%m%dT%H%M%SZ")
def slugify(value: str) -> str:
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
return slug or "turn"
def load_text(path: Path) -> str:
return path.read_text(encoding="utf-8")
def save_text(path: Path, content: str) -> None:
path.write_text(content.rstrip() + "\n", encoding="utf-8")
def insert_into_section(document: str, heading: str, block: str) -> str:
lines = document.splitlines()
try:
start = lines.index(heading)
except ValueError as exc:
raise SystemExit(f"heading not found: {heading}") from exc
end = len(lines)
for idx in range(start + 1, len(lines)):
if lines[idx].startswith("## "):
end = idx
break
insert_at = end
return "\n".join(lines[:insert_at] + [block] + lines[insert_at:]) + "\n"
def append_archive_line(section: str, line: str) -> None:
content = load_text(ARCHIVE_PATH)
updated = insert_into_section(content, ARCHIVE_SECTION[section], line)
save_text(ARCHIVE_PATH, updated)
def read_backlog_lines() -> list[str]:
return load_text(BACKLOG_PATH).splitlines()
def write_backlog_lines(lines: list[str]) -> None:
save_text(BACKLOG_PATH, "\n".join(lines))
def next_backlog_id(lane: str) -> str:
prefix = LANE_PREFIX[lane]
pattern = re.compile(rf"\[{re.escape(prefix)}(\d+)\]")
highest = 0
for line in read_backlog_lines():
match = pattern.search(line)
if match:
highest = max(highest, int(match.group(1)))
return f"{prefix}{highest + 1:03d}"
def backlog_entry_map() -> dict[str, str]:
entries: dict[str, str] = {}
pattern = re.compile(r"- \[([A-Z]\d+)\] (.+)")
for line in read_backlog_lines():
match = pattern.match(line)
if match:
entries[match.group(1)] = match.group(2)
return entries
def remove_backlog_ids(ids: list[str]) -> None:
id_set = set(ids)
pattern = re.compile(r"- \[([A-Z]\d+)\] ")
kept: list[str] = []
for line in read_backlog_lines():
match = pattern.match(line)
if match and match.group(1) in id_set:
continue
kept.append(line)
write_backlog_lines(kept)
def git_has_changes() -> bool:
result = subprocess.run(
["git", "-C", str(ROOT_DIR), "status", "--porcelain"],
check=True,
capture_output=True,
text=True,
)
return bool(result.stdout.strip())
def path_has_changes(paths: list[Path]) -> bool:
rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths]
result = subprocess.run(
["git", "-C", str(ROOT_DIR), "status", "--porcelain", "--", *rel_paths],
check=True,
capture_output=True,
text=True,
)
return bool(result.stdout.strip())
def git_commit(message: str, paths: list[Path]) -> None:
if not path_has_changes(paths):
return
rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths]
subprocess.run(["git", "-C", str(ROOT_DIR), "add", "--", *rel_paths], check=True)
subprocess.run(["git", "-C", str(ROOT_DIR), "commit", "-m", message], check=True)

9
scripts/workflow/install_hooks.sh Executable file
View file

@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
chmod +x "$ROOT_DIR/.githooks/commit-msg"
git -C "$ROOT_DIR" config core.hooksPath .githooks
echo "Installed tracked git hooks for $ROOT_DIR"

176
scripts/workflow/open_turn.py Executable file
View file

@ -0,0 +1,176 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
from common import (
BACKLOG_PATH,
ARCHIVE_PATH,
IMPLEMENTATION_PATH,
PROOF_PATH,
RESEARCH_ACTIVE_PATH,
append_archive_line,
backlog_entry_map,
git_commit,
remove_backlog_ids,
save_text,
slugify,
today_iso,
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Open a new implementation or research turn.")
parser.add_argument("--lane", required=True, choices=("implementation", "research"))
parser.add_argument("--title", required=True, help="Turn title.")
parser.add_argument("--summary", required=True, help="One-line proof or charter summary.")
parser.add_argument(
"--pick",
action="append",
default=[],
help="Backlog ID to pull into the turn. Repeat as needed.",
)
parser.add_argument(
"--commit",
action="store_true",
help="Commit the planning change automatically.",
)
parser.add_argument(
"--force",
action="store_true",
help="Replace an already-open turn instead of refusing.",
)
return parser.parse_args()
def render_selected(ids: list[str], items: dict[str, str]) -> str:
if not ids:
return "- none selected"
return "\n".join(f"- [{item_id}] {items[item_id]}" for item_id in ids)
def open_implementation_turn(title: str, summary: str, selected: str) -> None:
opened = today_iso()
save_text(
PROOF_PATH,
f"""# Implementation Proof: {title}
Status: open
Opened: {opened}
## Hypothesis
{summary}
## Scope
{selected}
## Non-goals
- unchanged from `THESIS.md` unless the user approves otherwise
## Definition of done
- current turn implementation is validated with direct evidence
- remaining fakes are listed plainly
""",
)
save_text(
IMPLEMENTATION_PATH,
f"""# Implementation Turn: {title}
Status: open
Opened: {opened}
## Goal
{summary}
## Selected backlog items
{selected}
## Notes
- Fill in the concrete implementation plan before coding if the live plan no longer matches the turn.
""",
)
def open_research_turn(title: str, summary: str, selected: str) -> None:
opened = today_iso()
save_text(
RESEARCH_ACTIVE_PATH,
f"""# Research Turn: {title}
Status: open
Opened: {opened}
## Charter
{summary}
## Selected backlog items
{selected}
## Hypothesis
TBD by the approved research turn.
## Dataset or source of truth
TBD by the approved research turn.
## Metrics
TBD by the approved research turn.
## Assumptions
TBD by the approved research turn.
## Falsification condition
TBD by the approved research turn.
""",
)
def main() -> None:
args = parse_args()
if args.lane == "implementation":
if "Status: open" in PROOF_PATH.read_text(encoding="utf-8") and not args.force:
raise SystemExit("implementation turn already open; close it first or pass --force")
else:
if "Status: open" in RESEARCH_ACTIVE_PATH.read_text(encoding="utf-8") and not args.force:
raise SystemExit("research turn already open; close it first or pass --force")
entries = backlog_entry_map()
missing = [item_id for item_id in args.pick if item_id not in entries]
if missing:
raise SystemExit(f"unknown backlog IDs: {', '.join(missing)}")
selected = render_selected(args.pick, entries)
if args.lane == "implementation":
open_implementation_turn(args.title, args.summary, selected)
else:
open_research_turn(args.title, args.summary, selected)
if args.pick:
remove_backlog_ids(args.pick)
append_archive_line(
"planning",
(
f"- {today_iso()}: opened {args.lane} turn `{slugify(args.title)}` "
f"from backlog items {', '.join(args.pick) if args.pick else 'none'}."
),
)
if args.commit:
commit_paths = [BACKLOG_PATH, ARCHIVE_PATH]
if args.lane == "implementation":
commit_paths.extend([PROOF_PATH, IMPLEMENTATION_PATH])
else:
commit_paths.append(RESEARCH_ACTIVE_PATH)
git_commit(
f"""Open {args.lane} turn: {args.title}
Proof: Establish the approved {args.lane} turn and move selected backlog items into active scope.
Assumptions: The selected backlog items are the approved scope for this turn.
Still fake: Opening a turn changes planning state only; the work itself is not implemented yet.""",
paths=commit_paths,
)
if __name__ == "__main__":
main()

37
scripts/workflow/review_diff.sh Executable file
View file

@ -0,0 +1,37 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
PROMPT_FILE="$ROOT_DIR/workflow/REVIEW_PROMPT.md"
RANGE="${1:-HEAD~1}"
if [[ "${1:-}" == "--help" ]]; then
cat <<'EOF'
Usage: bash scripts/workflow/review_diff.sh [git-diff-range]
Examples:
bash scripts/workflow/review_diff.sh HEAD~1
bash scripts/workflow/review_diff.sh main...HEAD
EOF
exit 0
fi
cat <<EOF
# Review Bundle
## Diff range
\`$RANGE\`
## Diff
\`\`\`diff
EOF
git -C "$ROOT_DIR" diff "$RANGE"
cat <<'EOF'
```
## Prompt
EOF
cat "$PROMPT_FILE"

28
workflow/REVIEW_PROMPT.md Normal file
View file

@ -0,0 +1,28 @@
# Adversarial review prompt
You are reviewing changes for a trading system.
Your job is to attack the work, not to praise it.
Look for:
1. Fake progress.
- Did the change make the system more real, or only more elaborate?
- Real means contact with live data, validated persistence, verified replay, or explicit evidence.
2. Smuggled scope.
- Did the diff stay inside the active proof or research charter?
3. Unstated assumptions.
- Prices, fees, latency, freshness, slippage, API guarantees, retention, and failure modes.
4. Placeholder dressed as real.
- Hardcoded values, mocks, TODOs, dead branches, or unverifiable claims.
5. Missing failure handling.
- Focus on real trading-system failures, not abstract defensive-programming trivia.
For every issue:
- state what it is
- state why it matters for this system
- state whether it should be fixed now, removed, or marked as still fake
Do not praise style.
Do not suggest adjacent features.
Do not expand scope.