diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..486ccb0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,162 @@ +# unrip agent rules + +This repository is run with a data-first trading-system workflow. + +## Core rule +Every change must directly serve the currently approved turn. + +## What a turn is +A turn is the currently approved slice of work. There are two lanes: + +- Implementation turn: + - defined by `PROOF.md` and `IMPLEMENTATION.md` + - the job is to implement what `IMPLEMENTATION.md` says, validate it against `PROOF.md`, and keep working until the scoped fake paths are removed or explicitly recorded as remaining blockers +- Research turn: + - defined by `research/ACTIVE.md` + - the job is to produce evidence against the active research charter, not product shape + +`BACKLOG.md` is not the turn. +`ARCHIVE.md` is not the turn. +They are planning and history artifacts. + +## Read order by task +Use the smallest context needed for the current task. + +### For implementation work +Read only: + +1. `THESIS.md` +2. `PROOF.md` +3. `IMPLEMENTATION.md` + +Do not read `BACKLOG.md`, `ARCHIVE.md`, or `research/ACTIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, review history, or switch to research. + +### For research work +Read only: + +1. `THESIS.md` +2. `research/ACTIVE.md` + +Do not read `BACKLOG.md`, `ARCHIVE.md`, `PROOF.md`, or `IMPLEMENTATION.md` unless the user explicitly asks to re-plan, compare lanes, or promote research into implementation. + +### For planning, re-planning, or archiving +Read: + +1. `THESIS.md` +2. the live turn files for the lane being changed +3. `BACKLOG.md` +4. `ARCHIVE.md` + +## Hard constraints +- Do not invent or adopt a new roadmap on your own. +- Do not expand scope beyond the active implementation proof or research charter. +- No backlog generation instead of implementation. +- No scaffolding ahead of demonstrated need. +- Quote collection and analytics are first-class from day one. They are not a later add-on. +- Do not present scaffolding, dashboards, placeholders, or mock flows as product progress. +- State assumptions before coding when the environment, venue, chain, or source behavior is uncertain. +- Declare what is still fake in every commit. +- Do not read `BACKLOG.md` or `ARCHIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, or inspect history. +- If you discover adjacent work, add it to `BACKLOG.md` instead of absorbing it into the current turn. +- Changes that widen risk require explicit user approval: + - live funds + - secret creation or rotation + - permanent infrastructure spend + - long-running external jobs + - destructive data migrations +- The long-term thesis may be proposed, but `THESIS.md` must not be rewritten without explicit user approval. + +## Iteration archive rule +When the user says to plan the next iteration, next turn, implementation turn, or proof sprint: + +- first preserve the finished turn before drafting new planning docs +- do not rewrite the live turn files until the current turn has been archived +- use the existing repo workflow and archive locations: + - implementation turn snapshots go in `archive/implementation/` + - research turn snapshots go in `archive/research/` +- prefer the tracked scripts: + - `python3 scripts/workflow/close_turn.py ...` + - `python3 scripts/workflow/open_turn.py ...` + +Planning or archiving is the one time it is correct to read `BACKLOG.md` and `ARCHIVE.md`. + +## Planning inputs +- `iteration`, `implementation turn`, `proof sprint`, and `next turn` mean the same planning slice unless the user says otherwise. +- Use `BACKLOG.md` only while planning, re-planning, or archiving. Do not read it while implementing or validating the active turn unless the user explicitly asks. +- Select backlog items and bugs that belong together under one proof topic instead of mixing unrelated work into the same slice. +- When an item is pulled into the live turn, remove it from `BACKLOG.md` and record the planning event in `ARCHIVE.md`. + +## Planning quality bar +- `IMPLEMENTATION.md` must be detailed enough that coding does not depend on rediscovering the plan mid-turn. +- `IMPLEMENTATION.md` should cover the end-to-end system touched by the proof: ingest, pricing, inventory, persistence, strategy, execution, control surfaces, logging, validation, tests, failure modes, and important edge cases. +- `PROOF.md` must be specific and falsifiable: scope, non-goals, definition of done, validation evidence, failure conditions, and clear statements about what is real versus fake. +- Both planning documents should think through the whole operator workflow and full system path, not isolated file edits. + +## Turn lanes +- Implementation lane: + - Governed by `PROOF.md` and `IMPLEMENTATION.md`. + - Must make the shared live and historical data path more real unless the turn is explicitly marked as pure ops. + - “Doing the turn” means implementing `IMPLEMENTATION.md`, validating against `PROOF.md`, fixing what fails, and continuing until the definition of done is met or a hard blocker is reached. +- Research lane: + - Governed by `research/ACTIVE.md`. + - Must name the hypothesis, dataset, metric, assumptions, and falsification condition. + - Research output is evidence, not product shape. + +## Commit rules +Every non-merge commit message body must include: +- `Proof: ...` +- `Assumptions: ...` +- `Still fake: ...` + +Install the tracked hook before relying on this: + +```bash +bash scripts/workflow/install_hooks.sh +``` + +## Post-implementation loop +- After implementation and validation, expect the user to test the slice and suggest small fixes. +- Treat those small fixes as part of closing the same turn unless the user explicitly changes scope. + +## Bug-fix rule +- If a bug is found, fix it, inspect analogous or affected locations, and apply the needed follow-up fixes there too. +- No bug fix is done without a regression test. +- If a meaningful automated test cannot be added, stop and explain why instead of claiming the fix is complete. + +## Real progress vs fake progress +Real progress means the repository can do more of the active proof with validated evidence from real systems. In this repo that means things like: +- ingesting real NEAR Intents flow +- storing durable inventory, pricing, decision, and execution records +- producing a real decision from live data +- making a real execution attempt through repo-controlled code +- proving blocked-path safety with explicit evidence + +Fake progress includes: +- docs-only motion without stronger runtime truth +- speculative architecture not required by the active proof +- dashboards or control surfaces without the underlying real path +- placeholders or mocks presented as finished work +- invented data, unverifiable claims, or abstractions not yet required + +## Safety rules +- Prefer the smallest real implementation that proves the active turn works. +- Keep secrets out of git, docs, and chat history. +- Use self-hosted or directly controlled infrastructure by default. +- If something is fake or incomplete, say so plainly. + +## Review standard +Before claiming a turn is done, be able to state: +- what became more real +- what was validated against real data or systems +- what is still fake +- what was deliberately not built + +## Working rule +Within an approved turn, continue implementing and validating until: +- `PROOF.md` is satisfied for the active scope, or +- a hard blocker requires user input. + +Do not silently open the next turn yourself. +Do not stop at “the structure exists.” +Do not stop while the active turn still depends on hidden dummy paths. +If something remains fake at the end of the turn, name it plainly. diff --git a/ARCHIVE.md b/ARCHIVE.md new file mode 100644 index 0000000..bb4b813 --- /dev/null +++ b/ARCHIVE.md @@ -0,0 +1,16 @@ +# Archive Index + +This file records turn openings, closures, and archived snapshots. + +Legacy note: +- Work completed before `2026-04-01` predates this workflow and is not retroactively indexed here. + +## Implementation Turns + +- 2026-04-02: `first-non-mocked-tradeable-loop-for-one-pair` closed with status `passed`. A live active-pair quote flowed through funding, inventory sync, strategy, and real mainnet quote responses with durable history. +## Research Turns + +## Planning Events +- 2026-04-01: workflow files initialized for thesis, implementation proof, backlog, archive, and research lane. +- 2026-04-01: active implementation proof rewritten from durable-history scaffolding to the first executable trade loop for one pair. +- 2026-04-02: opened implementation turn `pre-credit-funding-visibility-and-operator-alerts` from backlog items O003, O004. diff --git a/BACKLOG.md b/BACKLOG.md new file mode 100644 index 0000000..3b308a6 --- /dev/null +++ b/BACKLOG.md @@ -0,0 +1,36 @@ +# Backlog + +This file is the candidate pool for future work. It is not the active plan. + +Rules: +- Add ideas here when they do not belong in the current turn. +- Promotion from backlog to an active turn is a separate planning step. +- `scripts/workflow/add_backlog.py` can append new items with stable IDs. + +## Implementation Candidates +- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs. +- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results. +- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions. +- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting. +- [I005] Import local EUR/BTC history on disk into research tables or files for replay and baseline checks. +- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks. +- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory. +- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state. +- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility. + +## Research Candidates +- [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair. +- [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold. +- [R003] Measure whether stale quotes correlate with worse execution quality or higher reject rates. +- [R004] Import and inspect the local historical EUR/BTC data to see how it can seed replay and backtests. +- [R005] Measure how long treasury funding takes from external transfer to credited internal inventory. + +## Ops Candidates +- [O001] PostgreSQL backup and retention plan for analytics and audit history. +- [O002] Signing-secret and NEAR Intents account management for real execution credentials. + +## Bugs +- [B001] The previous storage-only turn did not reach a tradeable loop and pulled the workflow toward scaffolding. +- [B002] No reference price source exists, so the system cannot estimate edge. +- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path. +- [B004] The prior plan assumed external hot wallets were on the trade hot path instead of pre-funded NEAR Intents inventory. diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md new file mode 100644 index 0000000..6489d49 --- /dev/null +++ b/IMPLEMENTATION.md @@ -0,0 +1,235 @@ +# Implementation Turn: pre-credit funding visibility and operator alerts + +Status: open +Opened: 2026-04-02 + +## Goal +Make the already-live BTC/EURe loop operationally understandable between external funding and spendable credit, and make critical stale or failed states queryable as durable alert records instead of transient logs. + +## Selected backlog items +- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions. +- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck. + +## Design rule +Keep the already-proven execution and inventory truth path intact: +- verifier and bridge credit remain the only spendable truth +- pre-credit visibility is additive observability +- alerts are additive operability + +## Event backbone +Retain Kafka as the backbone. Add only the minimal new topics required: + +- `ops.funding_observation` +- `ops.alert` + +These topics are append-only evidence streams, not control planes. + +## Durable store +Extend PostgreSQL with new append-only families: +- `funding_observations` +- `ops_alerts` + +If a current-state materialization is needed, derive it from append-only records or keep it as a clearly named snapshot table. Do not replace the append-only record. + +## Service changes + +### 1. `liquidity-manager` +Extend the existing treasury owner instead of inventing a broad new funding stack. + +New responsibilities: +- retain active funding handles by chain and asset +- poll configured chain observers for those handles +- emit `ops.funding_observation` +- correlate chain observations with bridge `recent_deposits` +- expose latest observations and credit-correlation state through `/state` + +Expected state shape additions: +- `funding_observations_by_handle` +- `latest_funding_observation_at` +- `uncredited_funding_total_by_asset` +- `credit_correlation` + +Control additions: +- `POST /refresh-funding-observations` +- optional `POST /pause-funding-observer` +- optional `POST /resume-funding-observer` + +Important implementation constraints: +- do not change withdrawal behavior +- do not reuse spendable inventory fields for pre-credit state +- keep BTC and EURe observation records in one shared schema + +### 2. `inventory-sync` +Keep current spendable accounting intact. + +Possible additions: +- read the latest funding observations +- expose a separate `pre_credit_inbound` or `funding_visibility` field in `/state` + +Hard rule: +- `spendable`, `pending_inbound`, and strategy-facing credited truth must not become looser + +### 3. `history-writer` +Consume and persist the new topics: +- `ops.funding_observation` +- `ops.alert` + +Expose through `/state`: +- latest funding-observation write time +- latest alert write time +- counts or offsets for the new topics + +Add query-friendly indexes for: +- `tx_hash` +- `funding_handle` +- `alert_code` +- `ingested_at` + +### 4. `ops-sentinel` or equivalent alert evaluator +Add one small service only if needed to keep alert logic separate and testable. + +Responsibilities: +- consume: + - `ref.market_price` + - `state.intent_inventory` + - `ops.liquidity_action` + - `ops.funding_observation` + - `exec.trade_result` +- evaluate policy windows for stale and stuck conditions +- emit `ops.alert` raise/clear transitions +- expose current alert state + +Preferred alert model: +- stable `alert_code` +- `status`: `raised` or `cleared` +- `severity` +- `reason` +- `first_raised_at` +- `last_evaluated_at` +- correlation IDs when available + +Minimal alert set for this turn: +- `reference_price_stale` +- `inventory_snapshot_stale` +- `funding_seen_unconfirmed` +- `funding_confirmed_credit_pending` +- `funding_stuck` +- `executor_submission_failed` + +Do not add Slack, email, or paging integrations in this turn unless required to prove the path. Durable alert records plus HTTP state are sufficient. + +## Chain observer plan + +### BTC +Must be the first-class proof path. + +Implementation expectations: +- configurable observer endpoint +- look up the configured BTC deposit address +- detect: + - mempool appearance when available + - confirmation count + - credited transition once bridge/verifier catches up + +Assumption to keep explicit in code: +- a chain observer can disappear or lag independently of the bridge + +### Gnosis / EURe +Nice-to-have within the same schema, but BTC is the proof-critical path. + +If included this turn: +- watch the configured deposit address for EURe token transfers +- represent observation state in the same event model + +## Record shapes + +### `ops.funding_observation` +Required fields: +- `funding_observation_id` +- `account_id` +- `asset_id` +- `chain` +- `funding_handle` +- `source` +- `tx_hash` +- `status` +- `amount` +- `confirmations` +- `first_seen_at` +- `last_seen_at` +- `credited_at` when known +- `bridge_deposit_tx_hash` when correlated + +### `ops.alert` +Required fields: +- `alert_event_id` +- `alert_code` +- `status` +- `severity` +- `reason` +- `service_scope` +- `pair` when relevant +- `asset_id` when relevant +- `tx_hash` when relevant +- `raised_at` +- `cleared_at` +- `details` + +## Control surface expectations + +### `liquidity-manager` +Must expose: +- active deposit handles +- latest pre-credit funding observations +- latest credit correlation +- whether the funding observer is healthy or paused + +### alert evaluator +Must expose: +- current active alerts +- latest cleared alerts +- per-alert evaluation timestamps +- pause state + +### `history-writer` +Must expose the new topic offsets and write status for funding observations and alerts. + +## Tests +Required automated coverage: +- BTC funding observation remains non-spendable before credit +- alert transitions raise then clear on recovered stale state +- funding observation correlates to a later credited deposit without losing the original tx hash +- executor failure produces an alert event + +If a meaningful automated test cannot be written for a subpath, stop and record why instead of hand-waving. + +## Validation plan +- Safe induced stale-price alert: + - pause `market-reference-ingest` + - wait past freshness window + - observe `reference_price_stale` + - resume and observe clear +- Safe induced stale-inventory alert: + - pause `inventory-sync` + - wait past freshness window + - observe `inventory_snapshot_stale` + - resume and observe clear +- Funding visibility proof: + - use a real deposit address + - observe pre-credit chain state before bridge credit where timing allows + - later observe credit correlation +- Executor failure alert proof: + - use a controlled non-destructive failure mode such as temporary relay endpoint override in a safe environment or a replayable failure fixture + - verify `executor_submission_failed` + +## Out of scope on purpose +- No new trading strategy +- No historical backtest engine +- No broad observability stack +- No polished dashboard frontend +- No automated treasury refills + +## Still fake at turn open +- Pre-credit funding visibility is still missing from the live cluster. +- Alert state is still mostly implicit in service logs and manual inspection. +- There is no durable operator-facing record yet for "funds are on the way but not spendable." diff --git a/PROOF.md b/PROOF.md new file mode 100644 index 0000000..e819332 --- /dev/null +++ b/PROOF.md @@ -0,0 +1,154 @@ +# Implementation Proof: pre-credit funding visibility and operator alerts + +Status: open +Opened: 2026-04-02 + +## Target outcome +The next turn is complete only when `unrip` can show operators the gap between "funds sent" and "funds spendable" with durable evidence, and can surface actionable alert state for the live loop without requiring log-diving or manual SQL every time something stalls. + +This turn does not expand the trade hot path. It makes the existing live system more explainable and more operable. + +## Hypothesis +`unrip` becomes materially safer to operate once it can: + +1. observe configured funding handles before NEAR Intents credit +2. persist chain-level funding observations separately from spendable inventory +3. link pre-credit observations to later bridge and verifier credit where possible +4. emit durable alert state for stale prices, stale inventory, stuck funding, and failed execution submissions +5. expose that state through the same small control surfaces and PostgreSQL audit trail as the rest of the system + +If pre-credit funding remains invisible, or alert state still lives only in transient logs, the live loop is still too opaque for routine funded operation. + +## Scope +- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions. +- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck. + +## Non-goals +- No new venue, pair, or strategy logic. +- No dashboard or polished UI. +- No automatic treasury actions or auto-refunding. +- No attempt to treat chain-level observations as spendable inventory. +- No change to the live execution arming model. + +## Source-of-truth rule +Spendable inventory remains the existing truth: +- bridge and verifier credit determine spendable balances +- chain-level observations are visibility only + +The new pre-credit path must never be allowed to make a direction tradable earlier than the verifier does. + +## Required runtime behavior + +### Funding visibility +- The system must know the currently active funding handles for BTC and EURe. +- For configured chains, it must watch those handles before NEAR Intents credit appears. +- It must distinguish at least these states where applicable: + - `SEEN_UNCONFIRMED` + - `SEEN_CONFIRMED` + - `CREDIT_PENDING` + - `CREDITED` + - `FAILED_OR_STUCK` +- BTC is the must-prove chain because that is where live funding latency was operationally visible. +- Gnosis support may share the same event model even if its confirmation behavior is simpler. + +### Alerts +- Alerts must be durable records, not only log lines. +- At minimum the system must raise and clear alert state for: + - stale reference price + - stale inventory snapshot + - funding seen but not credited within policy + - execution submission failure +- Alert transitions must be inspectable through HTTP state and PostgreSQL. + +## Service expectations + +### `liquidity-manager` +Must become the owner of chain-level funding observations because it already owns deposit handles and treasury state. + +It must: +- refresh and retain active funding handles +- ingest chain-level funding observations +- reconcile them against bridge deposit state +- publish durable funding-observation records +- expose current per-handle funding state + +It must not: +- mark funds spendable +- trade on pre-credit observations + +### `inventory-sync` +May surface pre-credit funding context, but only under a clearly separate non-spendable field. + +It must not: +- merge pre-credit observations into `spendable` + +### `history-writer` +Must persist the new record families: +- funding observations +- alert events +- optionally current alert snapshots if the implementation separates events from state + +### Alert evaluator +This may be a new small service or a tightly scoped extension of an existing one, but it must: +- evaluate staleness and stuck conditions from durable inputs +- emit durable alert events +- expose current alert state and the latest reasons + +No broad orchestration or dashboard service should be introduced just to satisfy this proof. + +## Required durable storage +PostgreSQL must store at least: +- funding observations before bridge credit +- alert events or alert snapshots +- enough timestamps and IDs to correlate: + - funding handle + - chain tx hash + - later bridge tx hash or deposit record + - resulting verifier credit snapshot when available + +Kafka remains the event backbone. + +## Required control surface +At minimum operators must be able to inspect: +- active funding handles +- latest pre-credit observations by handle +- confirmation depth or equivalent chain state when available +- whether a funding action is still pending credit +- current active alerts and their reasons + +If a new alert service exists, it must expose: +- `GET /healthz` +- `GET /state` +- `POST /pause` +- `POST /resume` + +## Definition of done +- The live cluster still runs the previously proven funded trade loop unchanged for spendable truth. +- At least one real funding handle is watched at chain level before bridge credit. +- For at least one real deposit path, the system records a pre-credit observation before or during confirmation and later records the credited state separately. +- PostgreSQL contains durable records for the pre-credit funding path and alert path. +- Operators can inspect current funding observations and current alerts through control APIs. +- A stale price or stale inventory condition can be induced safely and becomes a durable alert. +- A funding delay or manually injected stuck condition can be represented as a durable alert with explicit reason fields. +- A failed execution submission path is represented as an alert without inventing fake venue traffic. +- Tests cover: + - pre-credit observations staying non-spendable + - alert raise and clear transitions + - correlation of funding observation to later credit when identifiers are available + +## Failure conditions +- Funding observations exist only in logs and are not queryable later. +- Pre-credit observations leak into spendable inventory or strategy gating. +- Alerts cannot be queried as current state. +- The only proof of stuck funding is a human manually watching a block explorer. +- The implementation adds a dashboard shell without stronger runtime truth. + +## Current real +- The first funded BTC/EURe live loop is already proven: + - real quote ingest + - real reference pricing + - real credited inventory + - real strategy decisions + - real `quote_response` submissions + - durable event chain in PostgreSQL +- Portfolio metrics are now durably computed and exposed, but alerting and pre-credit funding visibility are still incomplete. diff --git a/THESIS.md b/THESIS.md new file mode 100644 index 0000000..bc87b53 --- /dev/null +++ b/THESIS.md @@ -0,0 +1,50 @@ +# unrip thesis + +## Purpose +Build a data-first trading system whose first-class artifact is trustworthy market and execution history. + +The bot is one consumer of that truth. Analytics and backtesting are not bolted on later; they are part of the product from the beginning. + +## Product nucleus +The nucleus of the system is one shared truth pipeline: + +1. observe live market and intent flow +2. persist raw and normalized events durably +3. replay the same history for analytics and backtests +4. score candidate actions from the same canonical data model +5. execute only behind explicit safety gates and full auditability + +## Architectural invariants +- Live decisions and historical analysis must share the same canonical event model whenever practical. +- Raw events are kept alongside normalized and derived records. +- Every important decision should be reproducible from stored inputs and explicit assumptions. +- Execution must not outpace observability. If the system cannot explain what happened, it is not ready to trade. +- Quote collection and analytics are core product work, not support work. + +## Near-term thesis +Near term, `unrip` should become a narrow but truthful trading-data and decision pipeline for one real pair on one venue. + +That means: +- real upstream data +- durable storage beyond transient bus retention +- replayable history +- measurable candidate decisions +- no pretending that execution is safe before the data and analysis path is trustworthy + +## Long-term thesis +Long term, the same system should support: +- automated trading on selected pairs +- analytics and backtesting from retained ground-truth data +- cross-chain and cross-asset execution routing + +## Non-goals right now +- polished operator UI before the data loop is truthful +- broad multi-venue coverage before one core loop is real +- strategy claims without named assumptions and falsification criteria +- live trading with real funds before paper or tightly gated execution is trustworthy + +## Approval boundaries +The agent may propose changes to the thesis, but the user must approve: +- changes to the core product definition +- changes that raise the risk class of the system +- changes that create lasting infra cost or operational burden diff --git a/WORKFLOW.md b/WORKFLOW.md new file mode 100644 index 0000000..8e64444 --- /dev/null +++ b/WORKFLOW.md @@ -0,0 +1,59 @@ +# Workflow + +This repository uses a small tracked workflow layer instead of a large agent orchestration system. + +## Files +- `THESIS.md`: stable product intent +- `PROOF.md`: active implementation proof +- `IMPLEMENTATION.md`: current implementation turn +- `research/ACTIVE.md`: active research charter +- `BACKLOG.md`: parked ideas and bugs +- `ARCHIVE.md`: turn history index +- `workflow/REVIEW_PROMPT.md`: adversarial review prompt + +## Install the tracked git hook + +```bash +bash scripts/workflow/install_hooks.sh +``` + +## Add a backlog item + +```bash +python3 scripts/workflow/add_backlog.py --lane implementation --summary "Reference-price service for active pair inputs" +python3 scripts/workflow/add_backlog.py --lane research --summary "Test whether implied pair rate diverges from external reference prices after fees" +``` + +## Open a new implementation turn + +```bash +python3 scripts/workflow/open_turn.py \ + --lane implementation \ + --title "first executable trade loop for one pair" \ + --summary "Add reference pricing, strategy, durable audit history, and a real execution path for the active pair." \ + --pick I001 \ + --pick I002 \ + --pick I003 \ + --pick I004 +``` + +Use `--commit` if you want the planning change committed automatically. + +## Close the current turn and archive it + +```bash +python3 scripts/workflow/close_turn.py \ + --lane implementation \ + --status passed \ + --summary "A live active-pair quote flowed through pricing, decision, and a real execution attempt with durable audit records." +``` + +The script copies the live turn files into `archive/implementation/` or `archive/research/`, updates `ARCHIVE.md`, and can make the archive commit with `--commit`. + +## Build a review bundle + +```bash +bash scripts/workflow/review_diff.sh HEAD~1 +``` + +That emits a Markdown bundle containing the diff plus the adversarial review prompt for a separate review-only agent session. diff --git a/archive/implementation/.gitkeep b/archive/implementation/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/archive/implementation/.gitkeep @@ -0,0 +1 @@ + diff --git a/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md b/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md new file mode 100644 index 0000000..b6f37e4 --- /dev/null +++ b/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-implementation.md @@ -0,0 +1,544 @@ +# Implementation Turn: first non-mocked tradeable loop for one pair + +Status: open +Opened: 2026-04-01 + +## Goal +Build the first full live vertical slice that can actually attempt a trade: + +1. observe a real NEAR Intents quote +2. enrich it with live external reference pricing +3. synchronize spendable inventory already credited inside NEAR Intents +4. evaluate it in a real strategy service +5. gate it by inventory and arm state +6. submit it through a real Near Intents executor +7. persist the full chain in PostgreSQL + +This turn is explicitly not a storage side-quest. Storage, control surfaces, analytics support, and treasury funding visibility are included because they are required to make the trade loop trustworthy. + +## Selected backlog items +- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs. +- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results. +- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions. +- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting. +- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks. +- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory. +- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state. +- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility. +- [B002] No reference price source exists, so the system cannot estimate edge. +- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path. +- [B004] The current plan assumed external hot wallets were on the hot trade path instead of pre-funded internal inventory. + +## Architectural shape + +### Event backbone +Kafka or Redpanda remains the backbone between services. + +Required topic set for this turn: +- `raw.near_intents.quote` +- `norm.swap_demand` +- `ref.market_price` +- `state.intent_inventory` +- `ops.liquidity_action` +- `decision.trade_decision` +- `cmd.execute_trade` +- `exec.trade_result` + +### Durable store +PostgreSQL is the first durable analytics and audit layer. + +Why: +- enough write throughput for current scope +- strong queryability for replay, inspection, and debugging +- simple to operate +- better fit than inventing a warehouse right now + +### Service split +The first real loop should be seven services: +- `near-intents-ingest` +- `market-reference-ingest` +- `inventory-sync` +- `liquidity-manager` +- `history-writer` +- `strategy-engine` +- `trade-executor` + +## Service-by-service responsibilities + +### 1. `near-intents-ingest` +Responsibilities: +- connect to the NEAR Intents quote stream +- filter or classify the active pair +- emit raw and normalized events +- expose runtime state + +Inputs: +- NEAR Intents websocket +- pair filter config and runtime override + +Outputs: +- `raw.near_intents.quote` +- `norm.swap_demand` + +Control surface: +- `GET /healthz` +- `GET /state` +- `GET /pair-filter` +- `PUT /pair-filter` +- `POST /pair-filter/reset` + +Important edge cases: +- websocket disconnect +- reconnect storm +- invalid JSON +- pair silent but connection healthy + +### 2. `market-reference-ingest` +Responsibilities: +- subscribe to Kraken BTC/EUR pricing +- poll CoinGecko for fallback or cross-check pricing +- derive fair BTC/EURe price for both trade directions +- publish reference-price events +- expose latest source state and freshness + +Inputs: +- Kraken stream +- CoinGecko HTTP API + +Outputs: +- `ref.market_price` + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /refresh` +- `POST /pause` +- `POST /resume` + +Required state: +- latest Kraken price +- latest CoinGecko price +- derived fair rate +- freshness age +- source health flags + +Important edge cases: +- Kraken disconnect +- CoinGecko timeout or rate limit +- both sources stale +- conflicting sources beyond tolerance + +Required behavior: +- if Kraken is down but CoinGecko is fresh, degrade according to policy and record fallback usage +- if both are stale, mark the service unhealthy for decisioning +- the first implementation must make the EURe/EUR pricing assumption explicit: + - either treat EURe as 1:1 with EUR and record that plainly + - or use a separate sanity source and record the mapping logic + +### 3. `inventory-sync` +Responsibilities: +- read current credited spendable inventory from NEAR Intents internal state +- distinguish credited balances from pending deposits and pending withdrawals +- publish current internal inventory state +- expose freshness and reconciliation state + +Inputs: +- NEAR Intents inventory or verifier surfaces +- liquidity-manager state when relevant + +Outputs: +- `state.intent_inventory` + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /refresh` +- `POST /pause` +- `POST /resume` + +Required state: +- spendable balances by asset +- pending inbound funding by asset +- pending outbound withdrawal by asset +- last sync time +- reconciliation status + +Important edge cases: +- internal inventory surface unavailable +- external deposit seen but not yet credited internally +- credited balance lower than expected after funding +- stale inventory snapshot + +Required behavior: +- only credited internal inventory counts as spendable +- pending treasury movements must remain non-spendable +- stale inventory state must be visible and actionable + +### 4. `liquidity-manager` +Responsibilities: +- request and track deposit addresses or equivalent funding handles for treasury assets +- track treasury funding actions from external wallets into NEAR Intents +- track withdrawals and rebalance actions +- expose current funding pipeline state +- publish auditable liquidity action records + +Inputs: +- treasury configuration +- external funding wallets +- NEAR Intents deposit or withdrawal surfaces + +Outputs: +- `ops.liquidity_action` + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /refresh` +- `POST /pause` +- `POST /resume` +- `POST /freeze-withdrawals` + +Required state: +- active deposit addresses or funding handles +- recent funding attempts +- pending credits +- recent withdrawals +- rebalance state + +Important edge cases: +- deposit address request failure +- external wallet funded but NEAR Intents credit delayed +- duplicate funding detection +- unsupported asset or chain mapping + +Required behavior: +- treasury actions must remain visible and auditable +- funding must be decoupled from the per-trade hot path + +### 5. `history-writer` +Responsibilities: +- consume the core topics +- write append-only rows into PostgreSQL +- preserve causality across quote, decision, and execution records +- expose lag and write state + +Inputs: +- all core Kafka topics + +Outputs: +- PostgreSQL rows + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /pause` +- `POST /resume` +- `POST /drain` + +Required stored record families: +- raw quotes +- normalized demand +- reference prices +- inventory snapshots +- liquidity actions +- trade decisions +- execute commands +- execution results + +Important edge cases: +- PostgreSQL unavailable +- duplicate deliveries from Kafka +- offset commit mismatch after partial write + +Required behavior: +- never claim success before the write is durable +- surface last committed offsets and last successful write time + +### 6. `strategy-engine` +Responsibilities: +- join latest demand with latest price and inventory state +- compute implied quote rate +- compare it to fair rate +- apply freshness thresholds +- apply the initial 2% gross edge threshold +- apply max-notional and inventory checks +- emit auditable decision events +- emit `cmd.execute_trade` only when all gates pass and the system is armed + +Inputs: +- `norm.swap_demand` +- `ref.market_price` +- `state.intent_inventory` + +Outputs: +- `decision.trade_decision` +- `cmd.execute_trade` + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /arm` +- `POST /disarm` +- `POST /pause` +- `POST /resume` +- `PUT /threshold` +- `PUT /limits` + +Required decision state: +- arm state +- current threshold +- current notional cap +- latest decision +- latest rejected decision reason +- per-reason skip counters + +Required decision record fields: +- `decision_id` +- `quote_id` +- `pair` +- `direction` +- `implied_rate` +- `reference_rate` +- `gross_edge_pct` +- `price_freshness_ms` +- `inventory_snapshot` +- `decision` +- `decision_reason` + +Important edge cases: +- stale prices +- no price yet +- no inventory yet +- insufficient spendable inventory +- pending deposit exists but is not yet credited +- quote already expired +- repeated quote IDs +- one direction affordable and the other not +- fresh deploy starts armed by mistake + +### 7. `trade-executor` +Responsibilities: +- consume `cmd.execute_trade` +- load the correct Near Intents signing authority +- perform the real Near Intents submission using pre-funded internal inventory +- preserve idempotency across retries and restarts +- emit `exec.trade_result` + +Inputs: +- `cmd.execute_trade` +- signing secrets +- optional latest inventory state + +Outputs: +- `exec.trade_result` + +Control surface: +- `GET /healthz` +- `GET /state` +- `POST /arm` +- `POST /disarm` +- `POST /pause` +- `POST /resume` +- `POST /drain` + +Required state: +- arm state +- last command seen +- last request sent +- last venue response +- in-flight command count +- completed command count +- error counters + +Important edge cases: +- duplicate command delivery +- crash after submission but before result publish +- venue reject +- timeout with unknown outcome +- insufficient internal inventory detected late +- secret missing or invalid +- signer not authorized for the intended NEAR Intents account + +Required behavior: +- never silently swallow a venue error +- always publish a result record for attempted commands +- make duplicate suppression durable +- never try to bridge or top up inventory during trade execution +- start disarmed by default on fresh deploy + +## Persistence design + +### Why PostgreSQL now +- current scale does not require a special time-series database +- rows are easier to inspect than a custom file format for this phase +- joins across decision, command, and result matter more right now than raw ingestion throughput +- treasury, inventory, and execution joins matter more than raw bus throughput at this stage + +### Minimum tables or equivalent record families +- `raw_near_intents_quotes` +- `swap_demand_events` +- `market_price_events` +- `intent_inventory_snapshots` +- `liquidity_actions` +- `trade_decisions` +- `execute_trade_commands` +- `trade_execution_results` + +### Query requirements +Must be able to answer: +- what was the latest fair price when this decision was made +- what spendable inventory existed when this decision was made +- what treasury funding actions were still pending +- why was this quote skipped +- what command was emitted for this quote +- what happened when the executor submitted it +- what credited internal inventory existed at the time + +## Control and stop semantics +Every service must support inspection. + +State endpoints must be sufficient to answer: +- is it healthy +- is it connected +- what is it currently using as input state +- what was the last successful action +- why is it blocked, if blocked +- whether the state is authoritative or stale + +Stopping must be explicit: +- `pause` means stop taking new work but keep process alive +- `drain` means finish in-flight work then stop cleanly +- `disarm` means remain live and observable but refuse side effects + +This matters because the operator must be able to halt strategy or execution without destroying the whole pipeline. + +## Logging requirements +All services use structured JSON logs. + +Stable top-level fields: +- `level` +- `service` +- `component` +- `event` +- `namespace` +- `venue` +- `topic` +- `pair` + +Additional body fields when relevant: +- `quote_id` +- `decision_id` +- `command_id` +- `execution_id` +- `inventory_id` +- `liquidity_action_id` +- `gross_edge_pct` +- `price_freshness_ms` + +Log when: +- source connection is lost or reestablished +- price source becomes stale +- inventory state becomes stale or recovers +- funding action is requested, seen, credited, delayed, failed, or frozen +- strategy rejects with a meaningful reason +- strategy arms or disarms +- executor arms or disarms +- command is submitted +- venue rejects or times out +- PostgreSQL disconnects or recovers +- control API changes state + +Do not log every healthy message by default. + +## Failure and failover behavior + +### Reference pricing +- Kraken failure with fresh CoinGecko: + - allowed only if fallback policy says yes + - decision records must note fallback source use +- both sources stale: + - strategy blocks all execution + +### Inventory state +- missing or stale inventory state: + - strategy may still emit a non-actionable rejected decision + - strategy must not emit `cmd.execute_trade` +- pending funding action: + - remains non-spendable + - must be visible in control state and durable records +- treasury action service down: + - already funded trading may continue if inventory-sync is fresh + - new funding or withdrawal operations must be blocked visibly + +### Persistence +- PostgreSQL unavailable: + - history-writer unhealthy + - system must surface that audit history is impaired + - if the user wants strict mode, strategy or executor may be blocked until persistence returns + +### Execution +- timeout with unknown venue outcome: + - emit explicit uncertain result + - preserve idempotency state for recovery +- restart after partial submission: + - executor must not blindly resubmit without checking prior state +- executor sees lower real inventory than strategy snapshot: + - emit explicit failure result + - do not attempt fallback treasury movement + +## Testing and validation plan + +### Unit tests +- implied-rate calculation for both directions +- explicit EURe/EUR pricing-basis test +- threshold checks +- stale-price blocking +- insufficient-inventory blocking +- pending-deposit-not-spendable blocking +- command emission only when armed +- idempotency transitions in executor + +### Integration tests +- pricing service emits canonical reference-price events +- inventory-sync emits credited vs pending inventory state correctly +- liquidity-manager records funding actions and status transitions +- strategy consumes demand, pricing, and inventory and emits decision plus command as expected +- history-writer persists linked records across topics +- executor publishes result records for success and failure paths + +### Runtime validation in cluster +- inspect each service `/state` +- verify live reference pricing updates +- verify live internal inventory state +- verify one treasury funding path from request or deposit tracking to credited inventory +- verify strategy and executor start disarmed on a fresh deploy +- observe a rejected decision due to block condition +- arm strategy and executor +- keep the first live max notional tiny, on the order of a few EURe, before any larger cap +- observe one command emission +- observe one real Near Intents execution attempt +- verify PostgreSQL contains the full linked chain + +## Deliberately rejected for this turn +- dashboards +- broad multi-pair abstractions +- ML training infrastructure +- broad backtest framework +- polished operator UI +- warehouse or lakehouse design + +## Expected deliverables +- `market-reference-ingest` +- `inventory-sync` +- `liquidity-manager` +- `history-writer` +- `strategy-engine` +- `trade-executor` +- PostgreSQL schema and migrations or equivalent setup +- updated Kafka topic creation and config +- shared control API pattern for all long-running services +- tests for strategy, inventory, persistence, and executor behavior +- docs for operations, arm or disarm flow, and inspection commands + +## Validation target +This turn is only complete when the deployed system can, through repo-controlled services alone, take one live active-pair quote, price it, verify credited internal inventory, decide on it, gate it, submit a real Near Intents execution attempt, and preserve the full record chain in PostgreSQL. diff --git a/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-proof.md b/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-proof.md new file mode 100644 index 0000000..68bef4b --- /dev/null +++ b/archive/implementation/20260402T230445Z-first-non-mocked-tradeable-loop-for-one-pair-proof.md @@ -0,0 +1,356 @@ +# Implementation Proof: first non-mocked tradeable loop for one pair + +Status: open +Opened: 2026-04-01 + +## Target outcome +The active turn is not complete when the repo merely observes demand or emits placeholder commands. It is complete only when the deployed system can make a real Near Intents trade attempt for the active BTC/Gnosis EURe pair using pre-funded inventory already credited inside NEAR Intents, with the full chain stored durably for later analytics and backtesting. + +## Hypothesis +`unrip` becomes materially more real once one live NEAR swap-demand event can move through this full chain without mocks: + +1. real quote and demand ingestion +2. live external reference pricing +3. synchronized spendable inventory state from NEAR Intents +4. auditable strategy decision +5. inventory and safety gating +6. real Near Intents execution attempt using credited internal inventory +7. durable storage of the full event chain + +If any of those links is still dummy, manual-only, or opaque after the fact, the proof is not achieved. + +## Active pair and trading rule +- Active pair: + - `nep141:btc.omft.near` + - `nep141:gnosis-0x420ca0f9b9b604ce0fd9c18ef134c705e5fa3430.omft.near` +- Both directions must be implemented in the same decision and execution pipeline: + - BTC -> EURe + - EURe -> BTC +- Runtime spendable inventory inside NEAR Intents decides which direction can actually fire. +- External Gnosis and BTC wallets are treasury and funding inputs, not the per-trade spend path. +- Pending deposits or withdrawals must not be treated as spendable inventory. +- Initial decision rule: + - use Kraken plus CoinGecko reference prices + - make the EURe/EUR pricing assumption explicit in code and decision records + - require a gross edge of at least 2% + - require fresh reference pricing + - require fresh internal inventory state + - require sufficient spendable source-asset inventory already credited inside NEAR Intents + - keep strategy and executor disarmed by default on deploy + - start with a very small max notional for the first live attempt + +## Required services + +### `near-intents-ingest` +Responsibilities: +- connect to the live NEAR Intents quote feed +- maintain the active pair filter +- publish raw venue quotes +- publish normalized swap-demand events +- expose current ingest state + +Must not: +- make trading decisions +- execute trades + +### `market-reference-ingest` +Responsibilities: +- maintain current reference pricing for BTC/EUR and mapped BTC/EURe fair value +- use Kraken as the primary fast path +- use CoinGecko as fallback or cross-check +- publish reference-price events +- expose latest price state, freshness, and source health + +Must not: +- emit trade commands + +### `inventory-sync` +Responsibilities: +- read current spendable inventory from the NEAR Intents internal ledger or equivalent supported inventory surface +- distinguish credited inventory from pending deposits and withdrawals +- publish or serve current internal inventory state +- expose freshness, last successful sync time, and reconciliation status + +Must not: +- initiate treasury movements +- make trading decisions + +### `liquidity-manager` +Responsibilities: +- request or manage deposit addresses for supported treasury assets and chains +- track funding actions from external treasury wallets into NEAR Intents +- track pending deposits, credited deposits, withdrawals, and rebalance actions +- expose the current funding and treasury state +- publish or record liquidity actions so they are auditable + +Must not: +- execute trading strategy +- spend inventory on the trade hot path + +### `history-writer` +Responsibilities: +- consume the core Kafka topics +- write append-only records into PostgreSQL +- preserve enough IDs and timestamps to reconstruct causality +- expose writer lag, last committed offsets, and write health + +Must not: +- serve as the message bus +- make trading decisions + +### `strategy-engine` +Responsibilities: +- consume normalized demand plus fresh reference prices +- consume fresh spendable inventory state +- compute implied rate from the quote +- compare quote rate against external reference pricing +- apply the initial 2% gross edge threshold +- apply freshness, arming, and notional checks +- apply inventory-aware direction gating +- emit auditable decision events +- emit `cmd.execute_trade` only when a decision is actionable + +Must not: +- hold private keys +- execute venue actions directly + +### `trade-executor` +Responsibilities: +- consume `cmd.execute_trade` +- load the correct Near Intents signing authority at runtime +- perform the actual Near Intents submission against pre-funded internal inventory +- preserve idempotency and duplicate suppression +- publish `exec.trade_result` +- expose current arm state, recent attempts, and last venue error + +Must not: +- invent trading logic +- bypass Kafka and manual safety gates +- perform ad hoc bridging or treasury funding on the trade hot path + +## Required durable storage +PostgreSQL is the first durable analytics and audit store. + +It must store at least: +- raw quotes +- normalized demand +- reference-price snapshots +- internal inventory snapshots +- liquidity and funding actions +- strategy decisions +- trade commands +- execution results +- treasury status and reconciliation metadata + +Why PostgreSQL first: +- fast enough for the current streaming volume +- queryable for inspection and analytics +- simple to operate in the current cluster +- good fit for append-only audit plus current-state views + +Kafka remains the streaming backbone. PostgreSQL is not the hot transport path. + +## Required control surface +Every long-running service in this turn must expose a small HTTP control surface. The point is not a polished UI. The point is operability. + +At minimum every service must expose: +- `GET /healthz` +- `GET /state` + +Services with runtime controls must also expose the relevant action endpoints: + +### `near-intents-ingest` +- inspect: + - pair filter + - connection state + - frames received + - published counts +- control: + - update pair filter + - disable or reset pair filter + +### `market-reference-ingest` +- inspect: + - latest Kraken price + - latest CoinGecko price + - derived fair price + - freshness age + - source health +- control: + - pause or resume polling or streaming + - trigger ad hoc refresh + +### `inventory-sync` +- inspect: + - latest spendable balances by asset + - pending deposits and withdrawals + - last sync time + - reconciliation status +- control: + - trigger sync + - pause or resume sync + +### `liquidity-manager` +- inspect: + - active deposit addresses + - recent funding actions + - pending deposits + - credited deposits + - recent withdrawals +- control: + - request or rotate deposit address where supported + - refresh treasury status + - pause or resume funding trackers + - disable withdrawals or rebalance actions + +### `history-writer` +- inspect: + - last persisted offsets by topic + - last write time + - error count + - database connectivity +- control: + - pause writes + - resume writes + - drain and stop cleanly + +### `strategy-engine` +- inspect: + - arm state + - active threshold + - latest reference snapshot used + - latest inventory snapshot used + - latest decisions and reasons + - skipped counts by reason +- control: + - arm or disarm + - pause or resume decisions + - change threshold + - set notional cap + +### `trade-executor` +- inspect: + - arm state + - last command received + - last venue response + - last error + - in-flight and completed command counts +- control: + - arm or disarm execution + - pause consumption + - drain and stop cleanly + +Stopping a service must mean a graceful stop or drain, not “kill the pod and hope.” + +## Logging and observability requirements +All services must emit structured JSON logs with stable fields. + +Stable fields: +- `level` +- `service` +- `component` +- `event` +- `namespace` +- `venue` +- `topic` +- `pair` + +High-cardinality IDs belong in the body, not labels: +- `quote_id` +- `command_id` +- `decision_id` +- `execution_id` + +What must be logged: +- connection loss and recovery +- source stale state +- inventory-sync stale state +- treasury funding action requested, credited, failed, or stuck +- invalid messages +- decision accepted and decision rejected with reason +- arm and disarm actions +- execution submitted, rejected, failed, recovered, completed +- PostgreSQL disconnect and recovery +- control API failures + +What should not be logged: +- per-message noise without state change +- full payload spam by default + +## Required edge cases and failure handling +- stale reference prices must block decisions +- stale or missing internal inventory state must block execution +- insufficient credited inventory must block the affected direction only +- pending deposits must not be counted as spendable inventory +- deposit address created but never funded must not be mistaken for liquidity +- external transfer observed but not yet credited in NEAR Intents must stay non-spendable +- credited inventory drift between inventory-sync and executor must hard-fail the command and publish a result +- Kraken down but CoinGecko healthy should degrade, not crash, if policy allows fallback +- both reference sources stale must block decisions +- PostgreSQL down must surface a hard health failure and stop claiming the system is trade-ready +- liquidity-manager failure must block new funding operations but must not silently stop already funded trading +- duplicate `cmd.execute_trade` must not cause duplicate venue submission +- executor restart after a partial submission must preserve idempotency behavior +- Near Intents API or RPC failures must become explicit `trade_result` failure records +- pair inactivity must not be mistaken for pipeline breakage + +## Definition of done +- The deployed cluster is running all required services for this loop. +- Live Kraken and CoinGecko reference prices are flowing and inspectable. +- Spendable inventory inside NEAR Intents is flowing and inspectable. +- At least one treasury funding path is documented and verified: + - deposit address or supported funding path created + - funding action observed + - credited inventory visible in internal state +- PostgreSQL contains the full event chain for at least one real quote path. +- Strategy decisions are non-dummy and include explicit reason fields with edge, freshness, and inventory context. +- Both trade directions are implemented in the same decision and execution path. +- At least one direction can be armed based on available credited inventory. +- A real Near Intents execution attempt can be triggered through the repo-controlled path. +- The resulting venue response is captured durably and inspectably. +- Each service exposes the required health and state endpoints, and controlled pause or arm semantics where appropriate. +- Fresh deploys start disarmed by default and require explicit operator arming before side effects. +- Validation includes not only happy path but blocked-path evidence: + - stale reference blocked + - insufficient inventory blocked + - pending-deposit-not-spendable blocked + - disarmed executor blocked + +## Current real +- Real NEAR Intents quote data is flowing into the cluster. +- The pair filter can be configured and changed at runtime. +- Raw and normalized events already exist in Redpanda topics. +- The app is deployable and observable in Kubernetes. +- Kafka already provides the event backbone. + +## Current fake or incomplete +- Reference pricing does not exist yet. +- Spendable internal inventory sync does not exist yet. +- Treasury funding or deposit management does not exist yet. +- Strategy decisions are still dummy placeholders. +- Execution is still dummy. +- The current executor is not a real Near Intents adapter. +- PostgreSQL is not yet part of the loop. +- Most services do not yet have their own control surfaces. +- Real signing and treasury secret handling is not yet wired. + +## Failure conditions +This proof fails if any of the following is true: +- a trade can only happen via manual shell steps or out-of-band operator intervention +- the system depends on ad hoc bridging or external-wallet spending on the trade hot path +- decision logic is still embedded in a dummy or placeholder service +- executor logic is still simulated +- a service cannot be inspected or paused cleanly at runtime +- the system cannot explain why a quote did or did not become a trade +- the stored history cannot reconstruct quote, pricing, decision, command, and result +- both directions are not implemented +- the system attempts to execute without credited internal source inventory + +## Expected validation evidence +- live `ref.market_price` events and current state from the pricing control API +- live internal inventory state from the inventory-sync control API +- PostgreSQL rows linking quote, reference, inventory snapshot, decision, command, and execution result +- a blocked decision caused by stale price or insufficient inventory +- a funding action that becomes credited inventory +- a successful command emission from the strategy service with a recorded edge above 2% +- one real Near Intents execution attempt with stored request and response metadata diff --git a/archive/research/.gitkeep b/archive/research/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/archive/research/.gitkeep @@ -0,0 +1 @@ + diff --git a/research/ACTIVE.md b/research/ACTIVE.md new file mode 100644 index 0000000..c66f72e --- /dev/null +++ b/research/ACTIVE.md @@ -0,0 +1,14 @@ +# Research Turn + +Status: idle + +No approved research turn is active yet. + +When opening one, capture: +- charter +- hypothesis +- dataset or source of truth +- metrics +- assumptions +- falsification condition +- expected artifact paths under `research/experiments/` diff --git a/research/QUEUE.md b/research/QUEUE.md new file mode 100644 index 0000000..ead821c --- /dev/null +++ b/research/QUEUE.md @@ -0,0 +1,9 @@ +# Research Queue + +This queue holds strategy and analysis questions that are not yet an active research turn. + +## Candidate charters +- Determine which Kraken symbols and CoinGecko asset IDs give a trustworthy pricing basis for the active pair. +- Measure how often the active pair's implied rate diverges from external reference prices after a simple fee model. +- Test whether stale quotes correlate with worse downstream execution quality or higher reject rates. +- Import the local historical EUR/BTC data and decide how it should seed replay and backtesting. diff --git a/scripts/workflow/add_backlog.py b/scripts/workflow/add_backlog.py new file mode 100755 index 0000000..93529b5 --- /dev/null +++ b/scripts/workflow/add_backlog.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse + +from common import BACKLOG_PATH, BACKLOG_SECTION, insert_into_section, load_text, next_backlog_id, save_text + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Append an item to BACKLOG.md.") + parser.add_argument( + "--lane", + required=True, + choices=("implementation", "research", "ops", "bug"), + help="Backlog section to append to.", + ) + parser.add_argument("--summary", required=True, help="One-line backlog summary.") + parser.add_argument( + "--priority", + default="soon", + choices=("now", "soon", "later"), + help="Coarse urgency label.", + ) + parser.add_argument( + "--tags", + default="", + help="Comma-separated tags stored inline for grepability.", + ) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + item_id = next_backlog_id(args.lane) + tags = f" tags={args.tags}" if args.tags else "" + entry = f"- [{item_id}] ({args.priority}) {args.summary}{tags}" + updated = insert_into_section(load_text(BACKLOG_PATH), BACKLOG_SECTION[args.lane], entry) + save_text(BACKLOG_PATH, updated) + print(item_id) + + +if __name__ == "__main__": + main() diff --git a/scripts/workflow/close_turn.py b/scripts/workflow/close_turn.py new file mode 100755 index 0000000..6be70a4 --- /dev/null +++ b/scripts/workflow/close_turn.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import re +import shutil +from pathlib import Path + +from common import ( + ARCHIVE_PATH, + IMPLEMENTATION_ARCHIVE_DIR, + IMPLEMENTATION_PATH, + PROOF_PATH, + RESEARCH_ACTIVE_PATH, + RESEARCH_ARCHIVE_DIR, + append_archive_line, + git_commit, + load_text, + save_text, + slugify, + timestamp_slug, + today_iso, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Archive and close the current implementation or research turn.") + parser.add_argument("--lane", required=True, choices=("implementation", "research")) + parser.add_argument( + "--status", + required=True, + choices=("passed", "failed", "paused", "abandoned"), + help="Outcome of the turn.", + ) + parser.add_argument("--summary", required=True, help="One-line closure summary.") + parser.add_argument( + "--commit", + action="store_true", + help="Commit the archive change automatically.", + ) + return parser.parse_args() + + +def title_from_content(content: str, prefix: str) -> str: + match = re.search(rf"^# {re.escape(prefix)}: (.+)$", content, flags=re.MULTILINE) + if not match: + raise SystemExit(f"could not find title for {prefix.lower()}") + return match.group(1).strip() + + +def close_implementation_turn(status: str, summary: str) -> tuple[str, list[Path]]: + proof_content = load_text(PROOF_PATH) + implementation_content = load_text(IMPLEMENTATION_PATH) + title = title_from_content(proof_content, "Implementation Proof") + slug = slugify(title) + stamp = timestamp_slug() + + IMPLEMENTATION_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True) + proof_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-proof.md" + implementation_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-implementation.md" + shutil.copyfile(PROOF_PATH, proof_archive) + shutil.copyfile(IMPLEMENTATION_PATH, implementation_archive) + + save_text( + PROOF_PATH, + """# Implementation Proof + +Status: idle + +No approved implementation proof is active yet. +""", + ) + save_text( + IMPLEMENTATION_PATH, + """# Implementation Turn + +Status: idle + +No approved implementation turn is active yet. +""", + ) + + append_archive_line( + "implementation", + f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}", + ) + return title, [proof_archive, implementation_archive] + + +def close_research_turn(status: str, summary: str) -> tuple[str, list[Path]]: + research_content = load_text(RESEARCH_ACTIVE_PATH) + title = title_from_content(research_content, "Research Turn") + slug = slugify(title) + stamp = timestamp_slug() + + RESEARCH_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True) + research_archive = RESEARCH_ARCHIVE_DIR / f"{stamp}-{slug}.md" + shutil.copyfile(RESEARCH_ACTIVE_PATH, research_archive) + + save_text( + RESEARCH_ACTIVE_PATH, + """# Research Turn + +Status: idle + +No approved research turn is active yet. +""", + ) + + append_archive_line( + "research", + f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}", + ) + return title, [research_archive] + + +def main() -> None: + args = parse_args() + if args.lane == "implementation" and "Status: idle" in load_text(PROOF_PATH): + raise SystemExit("no active implementation turn to close") + if args.lane == "research" and "Status: idle" in load_text(RESEARCH_ACTIVE_PATH): + raise SystemExit("no active research turn to close") + + if args.lane == "implementation": + title, archived_paths = close_implementation_turn(args.status, args.summary) + else: + title, archived_paths = close_research_turn(args.status, args.summary) + + if args.commit: + paths = [ARCHIVE_PATH] + if args.lane == "implementation": + paths.extend([PROOF_PATH, IMPLEMENTATION_PATH]) + else: + paths.append(RESEARCH_ACTIVE_PATH) + paths.extend(archived_paths) + git_commit( + f"""Archive {args.lane} turn: {title} + +Proof: Preserve the completed {args.lane} turn and record its outcome in the tracked archive. +Assumptions: The archived files capture the relevant planning state for the completed turn. +Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.""", + paths=paths, + ) + + for archived_path in archived_paths: + print(archived_path.relative_to(RESEARCH_ACTIVE_PATH.parent.parent)) + + +if __name__ == "__main__": + main() diff --git a/scripts/workflow/common.py b/scripts/workflow/common.py new file mode 100755 index 0000000..831107b --- /dev/null +++ b/scripts/workflow/common.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import re +import subprocess +from datetime import datetime, timezone +from pathlib import Path + +ROOT_DIR = Path(__file__).resolve().parents[2] +BACKLOG_PATH = ROOT_DIR / "BACKLOG.md" +ARCHIVE_PATH = ROOT_DIR / "ARCHIVE.md" +PROOF_PATH = ROOT_DIR / "PROOF.md" +IMPLEMENTATION_PATH = ROOT_DIR / "IMPLEMENTATION.md" +RESEARCH_ACTIVE_PATH = ROOT_DIR / "research/ACTIVE.md" +IMPLEMENTATION_ARCHIVE_DIR = ROOT_DIR / "archive/implementation" +RESEARCH_ARCHIVE_DIR = ROOT_DIR / "archive/research" + +LANE_PREFIX = { + "implementation": "I", + "research": "R", + "ops": "O", + "bug": "B", +} + +BACKLOG_SECTION = { + "implementation": "## Implementation Candidates", + "research": "## Research Candidates", + "ops": "## Ops Candidates", + "bug": "## Bugs", +} + +ARCHIVE_SECTION = { + "implementation": "## Implementation Turns", + "research": "## Research Turns", + "planning": "## Planning Events", +} + + +def now_utc() -> datetime: + return datetime.now(timezone.utc) + + +def today_iso() -> str: + return now_utc().date().isoformat() + + +def timestamp_slug() -> str: + return now_utc().strftime("%Y%m%dT%H%M%SZ") + + +def slugify(value: str) -> str: + slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-") + return slug or "turn" + + +def load_text(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def save_text(path: Path, content: str) -> None: + path.write_text(content.rstrip() + "\n", encoding="utf-8") + + +def insert_into_section(document: str, heading: str, block: str) -> str: + lines = document.splitlines() + try: + start = lines.index(heading) + except ValueError as exc: + raise SystemExit(f"heading not found: {heading}") from exc + + end = len(lines) + for idx in range(start + 1, len(lines)): + if lines[idx].startswith("## "): + end = idx + break + + insert_at = end + return "\n".join(lines[:insert_at] + [block] + lines[insert_at:]) + "\n" + + +def append_archive_line(section: str, line: str) -> None: + content = load_text(ARCHIVE_PATH) + updated = insert_into_section(content, ARCHIVE_SECTION[section], line) + save_text(ARCHIVE_PATH, updated) + + +def read_backlog_lines() -> list[str]: + return load_text(BACKLOG_PATH).splitlines() + + +def write_backlog_lines(lines: list[str]) -> None: + save_text(BACKLOG_PATH, "\n".join(lines)) + + +def next_backlog_id(lane: str) -> str: + prefix = LANE_PREFIX[lane] + pattern = re.compile(rf"\[{re.escape(prefix)}(\d+)\]") + highest = 0 + for line in read_backlog_lines(): + match = pattern.search(line) + if match: + highest = max(highest, int(match.group(1))) + return f"{prefix}{highest + 1:03d}" + + +def backlog_entry_map() -> dict[str, str]: + entries: dict[str, str] = {} + pattern = re.compile(r"- \[([A-Z]\d+)\] (.+)") + for line in read_backlog_lines(): + match = pattern.match(line) + if match: + entries[match.group(1)] = match.group(2) + return entries + + +def remove_backlog_ids(ids: list[str]) -> None: + id_set = set(ids) + pattern = re.compile(r"- \[([A-Z]\d+)\] ") + kept: list[str] = [] + for line in read_backlog_lines(): + match = pattern.match(line) + if match and match.group(1) in id_set: + continue + kept.append(line) + write_backlog_lines(kept) + + +def git_has_changes() -> bool: + result = subprocess.run( + ["git", "-C", str(ROOT_DIR), "status", "--porcelain"], + check=True, + capture_output=True, + text=True, + ) + return bool(result.stdout.strip()) + + +def path_has_changes(paths: list[Path]) -> bool: + rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths] + result = subprocess.run( + ["git", "-C", str(ROOT_DIR), "status", "--porcelain", "--", *rel_paths], + check=True, + capture_output=True, + text=True, + ) + return bool(result.stdout.strip()) + + +def git_commit(message: str, paths: list[Path]) -> None: + if not path_has_changes(paths): + return + rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths] + subprocess.run(["git", "-C", str(ROOT_DIR), "add", "--", *rel_paths], check=True) + subprocess.run(["git", "-C", str(ROOT_DIR), "commit", "-m", message], check=True) diff --git a/scripts/workflow/install_hooks.sh b/scripts/workflow/install_hooks.sh new file mode 100755 index 0000000..530ceb5 --- /dev/null +++ b/scripts/workflow/install_hooks.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)" + +chmod +x "$ROOT_DIR/.githooks/commit-msg" +git -C "$ROOT_DIR" config core.hooksPath .githooks + +echo "Installed tracked git hooks for $ROOT_DIR" diff --git a/scripts/workflow/open_turn.py b/scripts/workflow/open_turn.py new file mode 100755 index 0000000..5d40ee7 --- /dev/null +++ b/scripts/workflow/open_turn.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse + +from common import ( + BACKLOG_PATH, + ARCHIVE_PATH, + IMPLEMENTATION_PATH, + PROOF_PATH, + RESEARCH_ACTIVE_PATH, + append_archive_line, + backlog_entry_map, + git_commit, + remove_backlog_ids, + save_text, + slugify, + today_iso, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Open a new implementation or research turn.") + parser.add_argument("--lane", required=True, choices=("implementation", "research")) + parser.add_argument("--title", required=True, help="Turn title.") + parser.add_argument("--summary", required=True, help="One-line proof or charter summary.") + parser.add_argument( + "--pick", + action="append", + default=[], + help="Backlog ID to pull into the turn. Repeat as needed.", + ) + parser.add_argument( + "--commit", + action="store_true", + help="Commit the planning change automatically.", + ) + parser.add_argument( + "--force", + action="store_true", + help="Replace an already-open turn instead of refusing.", + ) + return parser.parse_args() + + +def render_selected(ids: list[str], items: dict[str, str]) -> str: + if not ids: + return "- none selected" + return "\n".join(f"- [{item_id}] {items[item_id]}" for item_id in ids) + + +def open_implementation_turn(title: str, summary: str, selected: str) -> None: + opened = today_iso() + save_text( + PROOF_PATH, + f"""# Implementation Proof: {title} + +Status: open +Opened: {opened} + +## Hypothesis +{summary} + +## Scope +{selected} + +## Non-goals +- unchanged from `THESIS.md` unless the user approves otherwise + +## Definition of done +- current turn implementation is validated with direct evidence +- remaining fakes are listed plainly +""", + ) + save_text( + IMPLEMENTATION_PATH, + f"""# Implementation Turn: {title} + +Status: open +Opened: {opened} + +## Goal +{summary} + +## Selected backlog items +{selected} + +## Notes +- Fill in the concrete implementation plan before coding if the live plan no longer matches the turn. +""", + ) + + +def open_research_turn(title: str, summary: str, selected: str) -> None: + opened = today_iso() + save_text( + RESEARCH_ACTIVE_PATH, + f"""# Research Turn: {title} + +Status: open +Opened: {opened} + +## Charter +{summary} + +## Selected backlog items +{selected} + +## Hypothesis +TBD by the approved research turn. + +## Dataset or source of truth +TBD by the approved research turn. + +## Metrics +TBD by the approved research turn. + +## Assumptions +TBD by the approved research turn. + +## Falsification condition +TBD by the approved research turn. +""", + ) + + +def main() -> None: + args = parse_args() + if args.lane == "implementation": + if "Status: open" in PROOF_PATH.read_text(encoding="utf-8") and not args.force: + raise SystemExit("implementation turn already open; close it first or pass --force") + else: + if "Status: open" in RESEARCH_ACTIVE_PATH.read_text(encoding="utf-8") and not args.force: + raise SystemExit("research turn already open; close it first or pass --force") + + entries = backlog_entry_map() + missing = [item_id for item_id in args.pick if item_id not in entries] + if missing: + raise SystemExit(f"unknown backlog IDs: {', '.join(missing)}") + + selected = render_selected(args.pick, entries) + + if args.lane == "implementation": + open_implementation_turn(args.title, args.summary, selected) + else: + open_research_turn(args.title, args.summary, selected) + + if args.pick: + remove_backlog_ids(args.pick) + + append_archive_line( + "planning", + ( + f"- {today_iso()}: opened {args.lane} turn `{slugify(args.title)}` " + f"from backlog items {', '.join(args.pick) if args.pick else 'none'}." + ), + ) + + if args.commit: + commit_paths = [BACKLOG_PATH, ARCHIVE_PATH] + if args.lane == "implementation": + commit_paths.extend([PROOF_PATH, IMPLEMENTATION_PATH]) + else: + commit_paths.append(RESEARCH_ACTIVE_PATH) + git_commit( + f"""Open {args.lane} turn: {args.title} + +Proof: Establish the approved {args.lane} turn and move selected backlog items into active scope. +Assumptions: The selected backlog items are the approved scope for this turn. +Still fake: Opening a turn changes planning state only; the work itself is not implemented yet.""", + paths=commit_paths, + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/workflow/review_diff.sh b/scripts/workflow/review_diff.sh new file mode 100755 index 0000000..a62f072 --- /dev/null +++ b/scripts/workflow/review_diff.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)" +PROMPT_FILE="$ROOT_DIR/workflow/REVIEW_PROMPT.md" +RANGE="${1:-HEAD~1}" + +if [[ "${1:-}" == "--help" ]]; then + cat <<'EOF' +Usage: bash scripts/workflow/review_diff.sh [git-diff-range] + +Examples: + bash scripts/workflow/review_diff.sh HEAD~1 + bash scripts/workflow/review_diff.sh main...HEAD +EOF + exit 0 +fi + +cat <