Archive first live trade loop and open funding visibility turn
Proof: Preserve the completed first live BTC/EURe trade loop and establish the next approved implementation proof around pre-credit funding visibility and operator alerts. Assumptions: The live-trade loop is sufficiently proven by the recorded deposits, withdrawals, durable command/result chain, and successful mainnet quote responses; the next highest-value slice is operational visibility rather than new execution breadth. Still fake: The newly opened funding-visibility and alert turn is planning only; no pre-credit watcher or durable alert evaluator is implemented yet.
This commit is contained in:
philipp
parent
16e7b79978
commit
54dc05a94c
20 changed files with 2234 additions and 0 deletions
162
AGENTS.md
Normal file
162
AGENTS.md
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
# unrip agent rules
|
||||
|
||||
This repository is run with a data-first trading-system workflow.
|
||||
|
||||
## Core rule
|
||||
Every change must directly serve the currently approved turn.
|
||||
|
||||
## What a turn is
|
||||
A turn is the currently approved slice of work. There are two lanes:
|
||||
|
||||
- Implementation turn:
|
||||
- defined by `PROOF.md` and `IMPLEMENTATION.md`
|
||||
- the job is to implement what `IMPLEMENTATION.md` says, validate it against `PROOF.md`, and keep working until the scoped fake paths are removed or explicitly recorded as remaining blockers
|
||||
- Research turn:
|
||||
- defined by `research/ACTIVE.md`
|
||||
- the job is to produce evidence against the active research charter, not product shape
|
||||
|
||||
`BACKLOG.md` is not the turn.
|
||||
`ARCHIVE.md` is not the turn.
|
||||
They are planning and history artifacts.
|
||||
|
||||
## Read order by task
|
||||
Use the smallest context needed for the current task.
|
||||
|
||||
### For implementation work
|
||||
Read only:
|
||||
|
||||
1. `THESIS.md`
|
||||
2. `PROOF.md`
|
||||
3. `IMPLEMENTATION.md`
|
||||
|
||||
Do not read `BACKLOG.md`, `ARCHIVE.md`, or `research/ACTIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, review history, or switch to research.
|
||||
|
||||
### For research work
|
||||
Read only:
|
||||
|
||||
1. `THESIS.md`
|
||||
2. `research/ACTIVE.md`
|
||||
|
||||
Do not read `BACKLOG.md`, `ARCHIVE.md`, `PROOF.md`, or `IMPLEMENTATION.md` unless the user explicitly asks to re-plan, compare lanes, or promote research into implementation.
|
||||
|
||||
### For planning, re-planning, or archiving
|
||||
Read:
|
||||
|
||||
1. `THESIS.md`
|
||||
2. the live turn files for the lane being changed
|
||||
3. `BACKLOG.md`
|
||||
4. `ARCHIVE.md`
|
||||
|
||||
## Hard constraints
|
||||
- Do not invent or adopt a new roadmap on your own.
|
||||
- Do not expand scope beyond the active implementation proof or research charter.
|
||||
- No backlog generation instead of implementation.
|
||||
- No scaffolding ahead of demonstrated need.
|
||||
- Quote collection and analytics are first-class from day one. They are not a later add-on.
|
||||
- Do not present scaffolding, dashboards, placeholders, or mock flows as product progress.
|
||||
- State assumptions before coding when the environment, venue, chain, or source behavior is uncertain.
|
||||
- Declare what is still fake in every commit.
|
||||
- Do not read `BACKLOG.md` or `ARCHIVE.md` during an implementation turn unless the user explicitly asks to re-plan, open a new turn, or inspect history.
|
||||
- If you discover adjacent work, add it to `BACKLOG.md` instead of absorbing it into the current turn.
|
||||
- Changes that widen risk require explicit user approval:
|
||||
- live funds
|
||||
- secret creation or rotation
|
||||
- permanent infrastructure spend
|
||||
- long-running external jobs
|
||||
- destructive data migrations
|
||||
- The long-term thesis may be proposed, but `THESIS.md` must not be rewritten without explicit user approval.
|
||||
|
||||
## Iteration archive rule
|
||||
When the user says to plan the next iteration, next turn, implementation turn, or proof sprint:
|
||||
|
||||
- first preserve the finished turn before drafting new planning docs
|
||||
- do not rewrite the live turn files until the current turn has been archived
|
||||
- use the existing repo workflow and archive locations:
|
||||
- implementation turn snapshots go in `archive/implementation/`
|
||||
- research turn snapshots go in `archive/research/`
|
||||
- prefer the tracked scripts:
|
||||
- `python3 scripts/workflow/close_turn.py ...`
|
||||
- `python3 scripts/workflow/open_turn.py ...`
|
||||
|
||||
Planning or archiving is the one time it is correct to read `BACKLOG.md` and `ARCHIVE.md`.
|
||||
|
||||
## Planning inputs
|
||||
- `iteration`, `implementation turn`, `proof sprint`, and `next turn` mean the same planning slice unless the user says otherwise.
|
||||
- Use `BACKLOG.md` only while planning, re-planning, or archiving. Do not read it while implementing or validating the active turn unless the user explicitly asks.
|
||||
- Select backlog items and bugs that belong together under one proof topic instead of mixing unrelated work into the same slice.
|
||||
- When an item is pulled into the live turn, remove it from `BACKLOG.md` and record the planning event in `ARCHIVE.md`.
|
||||
|
||||
## Planning quality bar
|
||||
- `IMPLEMENTATION.md` must be detailed enough that coding does not depend on rediscovering the plan mid-turn.
|
||||
- `IMPLEMENTATION.md` should cover the end-to-end system touched by the proof: ingest, pricing, inventory, persistence, strategy, execution, control surfaces, logging, validation, tests, failure modes, and important edge cases.
|
||||
- `PROOF.md` must be specific and falsifiable: scope, non-goals, definition of done, validation evidence, failure conditions, and clear statements about what is real versus fake.
|
||||
- Both planning documents should think through the whole operator workflow and full system path, not isolated file edits.
|
||||
|
||||
## Turn lanes
|
||||
- Implementation lane:
|
||||
- Governed by `PROOF.md` and `IMPLEMENTATION.md`.
|
||||
- Must make the shared live and historical data path more real unless the turn is explicitly marked as pure ops.
|
||||
- “Doing the turn” means implementing `IMPLEMENTATION.md`, validating against `PROOF.md`, fixing what fails, and continuing until the definition of done is met or a hard blocker is reached.
|
||||
- Research lane:
|
||||
- Governed by `research/ACTIVE.md`.
|
||||
- Must name the hypothesis, dataset, metric, assumptions, and falsification condition.
|
||||
- Research output is evidence, not product shape.
|
||||
|
||||
## Commit rules
|
||||
Every non-merge commit message body must include:
|
||||
- `Proof: ...`
|
||||
- `Assumptions: ...`
|
||||
- `Still fake: ...`
|
||||
|
||||
Install the tracked hook before relying on this:
|
||||
|
||||
```bash
|
||||
bash scripts/workflow/install_hooks.sh
|
||||
```
|
||||
|
||||
## Post-implementation loop
|
||||
- After implementation and validation, expect the user to test the slice and suggest small fixes.
|
||||
- Treat those small fixes as part of closing the same turn unless the user explicitly changes scope.
|
||||
|
||||
## Bug-fix rule
|
||||
- If a bug is found, fix it, inspect analogous or affected locations, and apply the needed follow-up fixes there too.
|
||||
- No bug fix is done without a regression test.
|
||||
- If a meaningful automated test cannot be added, stop and explain why instead of claiming the fix is complete.
|
||||
|
||||
## Real progress vs fake progress
|
||||
Real progress means the repository can do more of the active proof with validated evidence from real systems. In this repo that means things like:
|
||||
- ingesting real NEAR Intents flow
|
||||
- storing durable inventory, pricing, decision, and execution records
|
||||
- producing a real decision from live data
|
||||
- making a real execution attempt through repo-controlled code
|
||||
- proving blocked-path safety with explicit evidence
|
||||
|
||||
Fake progress includes:
|
||||
- docs-only motion without stronger runtime truth
|
||||
- speculative architecture not required by the active proof
|
||||
- dashboards or control surfaces without the underlying real path
|
||||
- placeholders or mocks presented as finished work
|
||||
- invented data, unverifiable claims, or abstractions not yet required
|
||||
|
||||
## Safety rules
|
||||
- Prefer the smallest real implementation that proves the active turn works.
|
||||
- Keep secrets out of git, docs, and chat history.
|
||||
- Use self-hosted or directly controlled infrastructure by default.
|
||||
- If something is fake or incomplete, say so plainly.
|
||||
|
||||
## Review standard
|
||||
Before claiming a turn is done, be able to state:
|
||||
- what became more real
|
||||
- what was validated against real data or systems
|
||||
- what is still fake
|
||||
- what was deliberately not built
|
||||
|
||||
## Working rule
|
||||
Within an approved turn, continue implementing and validating until:
|
||||
- `PROOF.md` is satisfied for the active scope, or
|
||||
- a hard blocker requires user input.
|
||||
|
||||
Do not silently open the next turn yourself.
|
||||
Do not stop at “the structure exists.”
|
||||
Do not stop while the active turn still depends on hidden dummy paths.
|
||||
If something remains fake at the end of the turn, name it plainly.
|
||||
16
ARCHIVE.md
Normal file
16
ARCHIVE.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# Archive Index
|
||||
|
||||
This file records turn openings, closures, and archived snapshots.
|
||||
|
||||
Legacy note:
|
||||
- Work completed before `2026-04-01` predates this workflow and is not retroactively indexed here.
|
||||
|
||||
## Implementation Turns
|
||||
|
||||
- 2026-04-02: `first-non-mocked-tradeable-loop-for-one-pair` closed with status `passed`. A live active-pair quote flowed through funding, inventory sync, strategy, and real mainnet quote responses with durable history.
|
||||
## Research Turns
|
||||
|
||||
## Planning Events
|
||||
- 2026-04-01: workflow files initialized for thesis, implementation proof, backlog, archive, and research lane.
|
||||
- 2026-04-01: active implementation proof rewritten from durable-history scaffolding to the first executable trade loop for one pair.
|
||||
- 2026-04-02: opened implementation turn `pre-credit-funding-visibility-and-operator-alerts` from backlog items O003, O004.
|
||||
36
BACKLOG.md
Normal file
36
BACKLOG.md
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# Backlog
|
||||
|
||||
This file is the candidate pool for future work. It is not the active plan.
|
||||
|
||||
Rules:
|
||||
- Add ideas here when they do not belong in the current turn.
|
||||
- Promotion from backlog to an active turn is a separate planning step.
|
||||
- `scripts/workflow/add_backlog.py` can append new items with stable IDs.
|
||||
|
||||
## Implementation Candidates
|
||||
- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
|
||||
- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
|
||||
- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions.
|
||||
- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
|
||||
- [I005] Import local EUR/BTC history on disk into research tables or files for replay and baseline checks.
|
||||
- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
|
||||
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
|
||||
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
|
||||
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
|
||||
|
||||
## Research Candidates
|
||||
- [R001] Compare Kraken and CoinGecko drift and freshness for the assets needed to price the active pair.
|
||||
- [R002] Test whether the active pair's implied rate diverges from external reference prices enough to justify execution after a simple 2% gross threshold.
|
||||
- [R003] Measure whether stale quotes correlate with worse execution quality or higher reject rates.
|
||||
- [R004] Import and inspect the local historical EUR/BTC data to see how it can seed replay and backtests.
|
||||
- [R005] Measure how long treasury funding takes from external transfer to credited internal inventory.
|
||||
|
||||
## Ops Candidates
|
||||
- [O001] PostgreSQL backup and retention plan for analytics and audit history.
|
||||
- [O002] Signing-secret and NEAR Intents account management for real execution credentials.
|
||||
|
||||
## Bugs
|
||||
- [B001] The previous storage-only turn did not reach a tradeable loop and pulled the workflow toward scaffolding.
|
||||
- [B002] No reference price source exists, so the system cannot estimate edge.
|
||||
- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
|
||||
- [B004] The prior plan assumed external hot wallets were on the trade hot path instead of pre-funded NEAR Intents inventory.
|
||||
235
IMPLEMENTATION.md
Normal file
235
IMPLEMENTATION.md
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
# Implementation Turn: pre-credit funding visibility and operator alerts
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-02
|
||||
|
||||
## Goal
|
||||
Make the already-live BTC/EURe loop operationally understandable between external funding and spendable credit, and make critical stale or failed states queryable as durable alert records instead of transient logs.
|
||||
|
||||
## Selected backlog items
|
||||
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
|
||||
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
|
||||
|
||||
## Design rule
|
||||
Keep the already-proven execution and inventory truth path intact:
|
||||
- verifier and bridge credit remain the only spendable truth
|
||||
- pre-credit visibility is additive observability
|
||||
- alerts are additive operability
|
||||
|
||||
## Event backbone
|
||||
Retain Kafka as the backbone. Add only the minimal new topics required:
|
||||
|
||||
- `ops.funding_observation`
|
||||
- `ops.alert`
|
||||
|
||||
These topics are append-only evidence streams, not control planes.
|
||||
|
||||
## Durable store
|
||||
Extend PostgreSQL with new append-only families:
|
||||
- `funding_observations`
|
||||
- `ops_alerts`
|
||||
|
||||
If a current-state materialization is needed, derive it from append-only records or keep it as a clearly named snapshot table. Do not replace the append-only record.
|
||||
|
||||
## Service changes
|
||||
|
||||
### 1. `liquidity-manager`
|
||||
Extend the existing treasury owner instead of inventing a broad new funding stack.
|
||||
|
||||
New responsibilities:
|
||||
- retain active funding handles by chain and asset
|
||||
- poll configured chain observers for those handles
|
||||
- emit `ops.funding_observation`
|
||||
- correlate chain observations with bridge `recent_deposits`
|
||||
- expose latest observations and credit-correlation state through `/state`
|
||||
|
||||
Expected state shape additions:
|
||||
- `funding_observations_by_handle`
|
||||
- `latest_funding_observation_at`
|
||||
- `uncredited_funding_total_by_asset`
|
||||
- `credit_correlation`
|
||||
|
||||
Control additions:
|
||||
- `POST /refresh-funding-observations`
|
||||
- optional `POST /pause-funding-observer`
|
||||
- optional `POST /resume-funding-observer`
|
||||
|
||||
Important implementation constraints:
|
||||
- do not change withdrawal behavior
|
||||
- do not reuse spendable inventory fields for pre-credit state
|
||||
- keep BTC and EURe observation records in one shared schema
|
||||
|
||||
### 2. `inventory-sync`
|
||||
Keep current spendable accounting intact.
|
||||
|
||||
Possible additions:
|
||||
- read the latest funding observations
|
||||
- expose a separate `pre_credit_inbound` or `funding_visibility` field in `/state`
|
||||
|
||||
Hard rule:
|
||||
- `spendable`, `pending_inbound`, and strategy-facing credited truth must not become looser
|
||||
|
||||
### 3. `history-writer`
|
||||
Consume and persist the new topics:
|
||||
- `ops.funding_observation`
|
||||
- `ops.alert`
|
||||
|
||||
Expose through `/state`:
|
||||
- latest funding-observation write time
|
||||
- latest alert write time
|
||||
- counts or offsets for the new topics
|
||||
|
||||
Add query-friendly indexes for:
|
||||
- `tx_hash`
|
||||
- `funding_handle`
|
||||
- `alert_code`
|
||||
- `ingested_at`
|
||||
|
||||
### 4. `ops-sentinel` or equivalent alert evaluator
|
||||
Add one small service only if needed to keep alert logic separate and testable.
|
||||
|
||||
Responsibilities:
|
||||
- consume:
|
||||
- `ref.market_price`
|
||||
- `state.intent_inventory`
|
||||
- `ops.liquidity_action`
|
||||
- `ops.funding_observation`
|
||||
- `exec.trade_result`
|
||||
- evaluate policy windows for stale and stuck conditions
|
||||
- emit `ops.alert` raise/clear transitions
|
||||
- expose current alert state
|
||||
|
||||
Preferred alert model:
|
||||
- stable `alert_code`
|
||||
- `status`: `raised` or `cleared`
|
||||
- `severity`
|
||||
- `reason`
|
||||
- `first_raised_at`
|
||||
- `last_evaluated_at`
|
||||
- correlation IDs when available
|
||||
|
||||
Minimal alert set for this turn:
|
||||
- `reference_price_stale`
|
||||
- `inventory_snapshot_stale`
|
||||
- `funding_seen_unconfirmed`
|
||||
- `funding_confirmed_credit_pending`
|
||||
- `funding_stuck`
|
||||
- `executor_submission_failed`
|
||||
|
||||
Do not add Slack, email, or paging integrations in this turn unless required to prove the path. Durable alert records plus HTTP state are sufficient.
|
||||
|
||||
## Chain observer plan
|
||||
|
||||
### BTC
|
||||
Must be the first-class proof path.
|
||||
|
||||
Implementation expectations:
|
||||
- configurable observer endpoint
|
||||
- look up the configured BTC deposit address
|
||||
- detect:
|
||||
- mempool appearance when available
|
||||
- confirmation count
|
||||
- credited transition once bridge/verifier catches up
|
||||
|
||||
Assumption to keep explicit in code:
|
||||
- a chain observer can disappear or lag independently of the bridge
|
||||
|
||||
### Gnosis / EURe
|
||||
Nice-to-have within the same schema, but BTC is the proof-critical path.
|
||||
|
||||
If included this turn:
|
||||
- watch the configured deposit address for EURe token transfers
|
||||
- represent observation state in the same event model
|
||||
|
||||
## Record shapes
|
||||
|
||||
### `ops.funding_observation`
|
||||
Required fields:
|
||||
- `funding_observation_id`
|
||||
- `account_id`
|
||||
- `asset_id`
|
||||
- `chain`
|
||||
- `funding_handle`
|
||||
- `source`
|
||||
- `tx_hash`
|
||||
- `status`
|
||||
- `amount`
|
||||
- `confirmations`
|
||||
- `first_seen_at`
|
||||
- `last_seen_at`
|
||||
- `credited_at` when known
|
||||
- `bridge_deposit_tx_hash` when correlated
|
||||
|
||||
### `ops.alert`
|
||||
Required fields:
|
||||
- `alert_event_id`
|
||||
- `alert_code`
|
||||
- `status`
|
||||
- `severity`
|
||||
- `reason`
|
||||
- `service_scope`
|
||||
- `pair` when relevant
|
||||
- `asset_id` when relevant
|
||||
- `tx_hash` when relevant
|
||||
- `raised_at`
|
||||
- `cleared_at`
|
||||
- `details`
|
||||
|
||||
## Control surface expectations
|
||||
|
||||
### `liquidity-manager`
|
||||
Must expose:
|
||||
- active deposit handles
|
||||
- latest pre-credit funding observations
|
||||
- latest credit correlation
|
||||
- whether the funding observer is healthy or paused
|
||||
|
||||
### alert evaluator
|
||||
Must expose:
|
||||
- current active alerts
|
||||
- latest cleared alerts
|
||||
- per-alert evaluation timestamps
|
||||
- pause state
|
||||
|
||||
### `history-writer`
|
||||
Must expose the new topic offsets and write status for funding observations and alerts.
|
||||
|
||||
## Tests
|
||||
Required automated coverage:
|
||||
- BTC funding observation remains non-spendable before credit
|
||||
- alert transitions raise then clear on recovered stale state
|
||||
- funding observation correlates to a later credited deposit without losing the original tx hash
|
||||
- executor failure produces an alert event
|
||||
|
||||
If a meaningful automated test cannot be written for a subpath, stop and record why instead of hand-waving.
|
||||
|
||||
## Validation plan
|
||||
- Safe induced stale-price alert:
|
||||
- pause `market-reference-ingest`
|
||||
- wait past freshness window
|
||||
- observe `reference_price_stale`
|
||||
- resume and observe clear
|
||||
- Safe induced stale-inventory alert:
|
||||
- pause `inventory-sync`
|
||||
- wait past freshness window
|
||||
- observe `inventory_snapshot_stale`
|
||||
- resume and observe clear
|
||||
- Funding visibility proof:
|
||||
- use a real deposit address
|
||||
- observe pre-credit chain state before bridge credit where timing allows
|
||||
- later observe credit correlation
|
||||
- Executor failure alert proof:
|
||||
- use a controlled non-destructive failure mode such as temporary relay endpoint override in a safe environment or a replayable failure fixture
|
||||
- verify `executor_submission_failed`
|
||||
|
||||
## Out of scope on purpose
|
||||
- No new trading strategy
|
||||
- No historical backtest engine
|
||||
- No broad observability stack
|
||||
- No polished dashboard frontend
|
||||
- No automated treasury refills
|
||||
|
||||
## Still fake at turn open
|
||||
- Pre-credit funding visibility is still missing from the live cluster.
|
||||
- Alert state is still mostly implicit in service logs and manual inspection.
|
||||
- There is no durable operator-facing record yet for "funds are on the way but not spendable."
|
||||
154
PROOF.md
Normal file
154
PROOF.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# Implementation Proof: pre-credit funding visibility and operator alerts
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-02
|
||||
|
||||
## Target outcome
|
||||
The next turn is complete only when `unrip` can show operators the gap between "funds sent" and "funds spendable" with durable evidence, and can surface actionable alert state for the live loop without requiring log-diving or manual SQL every time something stalls.
|
||||
|
||||
This turn does not expand the trade hot path. It makes the existing live system more explainable and more operable.
|
||||
|
||||
## Hypothesis
|
||||
`unrip` becomes materially safer to operate once it can:
|
||||
|
||||
1. observe configured funding handles before NEAR Intents credit
|
||||
2. persist chain-level funding observations separately from spendable inventory
|
||||
3. link pre-credit observations to later bridge and verifier credit where possible
|
||||
4. emit durable alert state for stale prices, stale inventory, stuck funding, and failed execution submissions
|
||||
5. expose that state through the same small control surfaces and PostgreSQL audit trail as the rest of the system
|
||||
|
||||
If pre-credit funding remains invisible, or alert state still lives only in transient logs, the live loop is still too opaque for routine funded operation.
|
||||
|
||||
## Scope
|
||||
- [O003] Alerts for stale reference prices, stale inventory state, stuck funding actions, and failed executor submissions.
|
||||
- [O004] Pre-credit funding visibility for slow chains: watch configured deposit addresses at chain level, track inbound transfers through mempool and on-chain confirmation before bridge credit, persist that state separately from spendable inventory, and alert operators when funding is seen, delayed, or stuck.
|
||||
|
||||
## Non-goals
|
||||
- No new venue, pair, or strategy logic.
|
||||
- No dashboard or polished UI.
|
||||
- No automatic treasury actions or auto-refunding.
|
||||
- No attempt to treat chain-level observations as spendable inventory.
|
||||
- No change to the live execution arming model.
|
||||
|
||||
## Source-of-truth rule
|
||||
Spendable inventory remains the existing truth:
|
||||
- bridge and verifier credit determine spendable balances
|
||||
- chain-level observations are visibility only
|
||||
|
||||
The new pre-credit path must never be allowed to make a direction tradable earlier than the verifier does.
|
||||
|
||||
## Required runtime behavior
|
||||
|
||||
### Funding visibility
|
||||
- The system must know the currently active funding handles for BTC and EURe.
|
||||
- For configured chains, it must watch those handles before NEAR Intents credit appears.
|
||||
- It must distinguish at least these states where applicable:
|
||||
- `SEEN_UNCONFIRMED`
|
||||
- `SEEN_CONFIRMED`
|
||||
- `CREDIT_PENDING`
|
||||
- `CREDITED`
|
||||
- `FAILED_OR_STUCK`
|
||||
- BTC is the must-prove chain because that is where live funding latency was operationally visible.
|
||||
- Gnosis support may share the same event model even if its confirmation behavior is simpler.
|
||||
|
||||
### Alerts
|
||||
- Alerts must be durable records, not only log lines.
|
||||
- At minimum the system must raise and clear alert state for:
|
||||
- stale reference price
|
||||
- stale inventory snapshot
|
||||
- funding seen but not credited within policy
|
||||
- execution submission failure
|
||||
- Alert transitions must be inspectable through HTTP state and PostgreSQL.
|
||||
|
||||
## Service expectations
|
||||
|
||||
### `liquidity-manager`
|
||||
Must become the owner of chain-level funding observations because it already owns deposit handles and treasury state.
|
||||
|
||||
It must:
|
||||
- refresh and retain active funding handles
|
||||
- ingest chain-level funding observations
|
||||
- reconcile them against bridge deposit state
|
||||
- publish durable funding-observation records
|
||||
- expose current per-handle funding state
|
||||
|
||||
It must not:
|
||||
- mark funds spendable
|
||||
- trade on pre-credit observations
|
||||
|
||||
### `inventory-sync`
|
||||
May surface pre-credit funding context, but only under a clearly separate non-spendable field.
|
||||
|
||||
It must not:
|
||||
- merge pre-credit observations into `spendable`
|
||||
|
||||
### `history-writer`
|
||||
Must persist the new record families:
|
||||
- funding observations
|
||||
- alert events
|
||||
- optionally current alert snapshots if the implementation separates events from state
|
||||
|
||||
### Alert evaluator
|
||||
This may be a new small service or a tightly scoped extension of an existing one, but it must:
|
||||
- evaluate staleness and stuck conditions from durable inputs
|
||||
- emit durable alert events
|
||||
- expose current alert state and the latest reasons
|
||||
|
||||
No broad orchestration or dashboard service should be introduced just to satisfy this proof.
|
||||
|
||||
## Required durable storage
|
||||
PostgreSQL must store at least:
|
||||
- funding observations before bridge credit
|
||||
- alert events or alert snapshots
|
||||
- enough timestamps and IDs to correlate:
|
||||
- funding handle
|
||||
- chain tx hash
|
||||
- later bridge tx hash or deposit record
|
||||
- resulting verifier credit snapshot when available
|
||||
|
||||
Kafka remains the event backbone.
|
||||
|
||||
## Required control surface
|
||||
At minimum operators must be able to inspect:
|
||||
- active funding handles
|
||||
- latest pre-credit observations by handle
|
||||
- confirmation depth or equivalent chain state when available
|
||||
- whether a funding action is still pending credit
|
||||
- current active alerts and their reasons
|
||||
|
||||
If a new alert service exists, it must expose:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
|
||||
## Definition of done
|
||||
- The live cluster still runs the previously proven funded trade loop unchanged for spendable truth.
|
||||
- At least one real funding handle is watched at chain level before bridge credit.
|
||||
- For at least one real deposit path, the system records a pre-credit observation before or during confirmation and later records the credited state separately.
|
||||
- PostgreSQL contains durable records for the pre-credit funding path and alert path.
|
||||
- Operators can inspect current funding observations and current alerts through control APIs.
|
||||
- A stale price or stale inventory condition can be induced safely and becomes a durable alert.
|
||||
- A funding delay or manually injected stuck condition can be represented as a durable alert with explicit reason fields.
|
||||
- A failed execution submission path is represented as an alert without inventing fake venue traffic.
|
||||
- Tests cover:
|
||||
- pre-credit observations staying non-spendable
|
||||
- alert raise and clear transitions
|
||||
- correlation of funding observation to later credit when identifiers are available
|
||||
|
||||
## Failure conditions
|
||||
- Funding observations exist only in logs and are not queryable later.
|
||||
- Pre-credit observations leak into spendable inventory or strategy gating.
|
||||
- Alerts cannot be queried as current state.
|
||||
- The only proof of stuck funding is a human manually watching a block explorer.
|
||||
- The implementation adds a dashboard shell without stronger runtime truth.
|
||||
|
||||
## Current real
|
||||
- The first funded BTC/EURe live loop is already proven:
|
||||
- real quote ingest
|
||||
- real reference pricing
|
||||
- real credited inventory
|
||||
- real strategy decisions
|
||||
- real `quote_response` submissions
|
||||
- durable event chain in PostgreSQL
|
||||
- Portfolio metrics are now durably computed and exposed, but alerting and pre-credit funding visibility are still incomplete.
|
||||
50
THESIS.md
Normal file
50
THESIS.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
# unrip thesis
|
||||
|
||||
## Purpose
|
||||
Build a data-first trading system whose first-class artifact is trustworthy market and execution history.
|
||||
|
||||
The bot is one consumer of that truth. Analytics and backtesting are not bolted on later; they are part of the product from the beginning.
|
||||
|
||||
## Product nucleus
|
||||
The nucleus of the system is one shared truth pipeline:
|
||||
|
||||
1. observe live market and intent flow
|
||||
2. persist raw and normalized events durably
|
||||
3. replay the same history for analytics and backtests
|
||||
4. score candidate actions from the same canonical data model
|
||||
5. execute only behind explicit safety gates and full auditability
|
||||
|
||||
## Architectural invariants
|
||||
- Live decisions and historical analysis must share the same canonical event model whenever practical.
|
||||
- Raw events are kept alongside normalized and derived records.
|
||||
- Every important decision should be reproducible from stored inputs and explicit assumptions.
|
||||
- Execution must not outpace observability. If the system cannot explain what happened, it is not ready to trade.
|
||||
- Quote collection and analytics are core product work, not support work.
|
||||
|
||||
## Near-term thesis
|
||||
Near term, `unrip` should become a narrow but truthful trading-data and decision pipeline for one real pair on one venue.
|
||||
|
||||
That means:
|
||||
- real upstream data
|
||||
- durable storage beyond transient bus retention
|
||||
- replayable history
|
||||
- measurable candidate decisions
|
||||
- no pretending that execution is safe before the data and analysis path is trustworthy
|
||||
|
||||
## Long-term thesis
|
||||
Long term, the same system should support:
|
||||
- automated trading on selected pairs
|
||||
- analytics and backtesting from retained ground-truth data
|
||||
- cross-chain and cross-asset execution routing
|
||||
|
||||
## Non-goals right now
|
||||
- polished operator UI before the data loop is truthful
|
||||
- broad multi-venue coverage before one core loop is real
|
||||
- strategy claims without named assumptions and falsification criteria
|
||||
- live trading with real funds before paper or tightly gated execution is trustworthy
|
||||
|
||||
## Approval boundaries
|
||||
The agent may propose changes to the thesis, but the user must approve:
|
||||
- changes to the core product definition
|
||||
- changes that raise the risk class of the system
|
||||
- changes that create lasting infra cost or operational burden
|
||||
59
WORKFLOW.md
Normal file
59
WORKFLOW.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Workflow
|
||||
|
||||
This repository uses a small tracked workflow layer instead of a large agent orchestration system.
|
||||
|
||||
## Files
|
||||
- `THESIS.md`: stable product intent
|
||||
- `PROOF.md`: active implementation proof
|
||||
- `IMPLEMENTATION.md`: current implementation turn
|
||||
- `research/ACTIVE.md`: active research charter
|
||||
- `BACKLOG.md`: parked ideas and bugs
|
||||
- `ARCHIVE.md`: turn history index
|
||||
- `workflow/REVIEW_PROMPT.md`: adversarial review prompt
|
||||
|
||||
## Install the tracked git hook
|
||||
|
||||
```bash
|
||||
bash scripts/workflow/install_hooks.sh
|
||||
```
|
||||
|
||||
## Add a backlog item
|
||||
|
||||
```bash
|
||||
python3 scripts/workflow/add_backlog.py --lane implementation --summary "Reference-price service for active pair inputs"
|
||||
python3 scripts/workflow/add_backlog.py --lane research --summary "Test whether implied pair rate diverges from external reference prices after fees"
|
||||
```
|
||||
|
||||
## Open a new implementation turn
|
||||
|
||||
```bash
|
||||
python3 scripts/workflow/open_turn.py \
|
||||
--lane implementation \
|
||||
--title "first executable trade loop for one pair" \
|
||||
--summary "Add reference pricing, strategy, durable audit history, and a real execution path for the active pair." \
|
||||
--pick I001 \
|
||||
--pick I002 \
|
||||
--pick I003 \
|
||||
--pick I004
|
||||
```
|
||||
|
||||
Use `--commit` if you want the planning change committed automatically.
|
||||
|
||||
## Close the current turn and archive it
|
||||
|
||||
```bash
|
||||
python3 scripts/workflow/close_turn.py \
|
||||
--lane implementation \
|
||||
--status passed \
|
||||
--summary "A live active-pair quote flowed through pricing, decision, and a real execution attempt with durable audit records."
|
||||
```
|
||||
|
||||
The script copies the live turn files into `archive/implementation/` or `archive/research/`, updates `ARCHIVE.md`, and can make the archive commit with `--commit`.
|
||||
|
||||
## Build a review bundle
|
||||
|
||||
```bash
|
||||
bash scripts/workflow/review_diff.sh HEAD~1
|
||||
```
|
||||
|
||||
That emits a Markdown bundle containing the diff plus the adversarial review prompt for a separate review-only agent session.
|
||||
1
archive/implementation/.gitkeep
Normal file
1
archive/implementation/.gitkeep
Normal file
|
|
@ -0,0 +1 @@
|
|||
|
||||
|
|
@ -0,0 +1,544 @@
|
|||
# Implementation Turn: first non-mocked tradeable loop for one pair
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-01
|
||||
|
||||
## Goal
|
||||
Build the first full live vertical slice that can actually attempt a trade:
|
||||
|
||||
1. observe a real NEAR Intents quote
|
||||
2. enrich it with live external reference pricing
|
||||
3. synchronize spendable inventory already credited inside NEAR Intents
|
||||
4. evaluate it in a real strategy service
|
||||
5. gate it by inventory and arm state
|
||||
6. submit it through a real Near Intents executor
|
||||
7. persist the full chain in PostgreSQL
|
||||
|
||||
This turn is explicitly not a storage side-quest. Storage, control surfaces, analytics support, and treasury funding visibility are included because they are required to make the trade loop trustworthy.
|
||||
|
||||
## Selected backlog items
|
||||
- [I001] Hybrid reference-price service using Kraken stream and CoinGecko poll or fallback for the active pair inputs.
|
||||
- [I002] PostgreSQL event store plus history writer for quotes, reference prices, decisions, commands, and execution results.
|
||||
- [I003] Strategy engine that consumes `norm.swap_demand` plus reference prices and emits auditable decisions.
|
||||
- [I004] Real Near Intents executor service using pre-funded internal inventory, with explicit arming and idempotent result reporting.
|
||||
- [I006] Decision-to-command safety gate with explicit arm or disarm state, notional caps, and inventory freshness checks.
|
||||
- [I007] Inventory-aware execution rule: implement both directions, but only fire the side backed by credited internal source-asset inventory.
|
||||
- [I008] Inventory-sync service for NEAR Intents internal balances and pending funding state.
|
||||
- [I009] Liquidity-manager service for deposit addresses, funding actions, and treasury visibility.
|
||||
- [B002] No reference price source exists, so the system cannot estimate edge.
|
||||
- [B003] Dummy reactor and dummy executor prevent a non-mocked trade path.
|
||||
- [B004] The current plan assumed external hot wallets were on the hot trade path instead of pre-funded internal inventory.
|
||||
|
||||
## Architectural shape
|
||||
|
||||
### Event backbone
|
||||
Kafka or Redpanda remains the backbone between services.
|
||||
|
||||
Required topic set for this turn:
|
||||
- `raw.near_intents.quote`
|
||||
- `norm.swap_demand`
|
||||
- `ref.market_price`
|
||||
- `state.intent_inventory`
|
||||
- `ops.liquidity_action`
|
||||
- `decision.trade_decision`
|
||||
- `cmd.execute_trade`
|
||||
- `exec.trade_result`
|
||||
|
||||
### Durable store
|
||||
PostgreSQL is the first durable analytics and audit layer.
|
||||
|
||||
Why:
|
||||
- enough write throughput for current scope
|
||||
- strong queryability for replay, inspection, and debugging
|
||||
- simple to operate
|
||||
- better fit than inventing a warehouse right now
|
||||
|
||||
### Service split
|
||||
The first real loop should be seven services:
|
||||
- `near-intents-ingest`
|
||||
- `market-reference-ingest`
|
||||
- `inventory-sync`
|
||||
- `liquidity-manager`
|
||||
- `history-writer`
|
||||
- `strategy-engine`
|
||||
- `trade-executor`
|
||||
|
||||
## Service-by-service responsibilities
|
||||
|
||||
### 1. `near-intents-ingest`
|
||||
Responsibilities:
|
||||
- connect to the NEAR Intents quote stream
|
||||
- filter or classify the active pair
|
||||
- emit raw and normalized events
|
||||
- expose runtime state
|
||||
|
||||
Inputs:
|
||||
- NEAR Intents websocket
|
||||
- pair filter config and runtime override
|
||||
|
||||
Outputs:
|
||||
- `raw.near_intents.quote`
|
||||
- `norm.swap_demand`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `GET /pair-filter`
|
||||
- `PUT /pair-filter`
|
||||
- `POST /pair-filter/reset`
|
||||
|
||||
Important edge cases:
|
||||
- websocket disconnect
|
||||
- reconnect storm
|
||||
- invalid JSON
|
||||
- pair silent but connection healthy
|
||||
|
||||
### 2. `market-reference-ingest`
|
||||
Responsibilities:
|
||||
- subscribe to Kraken BTC/EUR pricing
|
||||
- poll CoinGecko for fallback or cross-check pricing
|
||||
- derive fair BTC/EURe price for both trade directions
|
||||
- publish reference-price events
|
||||
- expose latest source state and freshness
|
||||
|
||||
Inputs:
|
||||
- Kraken stream
|
||||
- CoinGecko HTTP API
|
||||
|
||||
Outputs:
|
||||
- `ref.market_price`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /refresh`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
|
||||
Required state:
|
||||
- latest Kraken price
|
||||
- latest CoinGecko price
|
||||
- derived fair rate
|
||||
- freshness age
|
||||
- source health flags
|
||||
|
||||
Important edge cases:
|
||||
- Kraken disconnect
|
||||
- CoinGecko timeout or rate limit
|
||||
- both sources stale
|
||||
- conflicting sources beyond tolerance
|
||||
|
||||
Required behavior:
|
||||
- if Kraken is down but CoinGecko is fresh, degrade according to policy and record fallback usage
|
||||
- if both are stale, mark the service unhealthy for decisioning
|
||||
- the first implementation must make the EURe/EUR pricing assumption explicit:
|
||||
- either treat EURe as 1:1 with EUR and record that plainly
|
||||
- or use a separate sanity source and record the mapping logic
|
||||
|
||||
### 3. `inventory-sync`
|
||||
Responsibilities:
|
||||
- read current credited spendable inventory from NEAR Intents internal state
|
||||
- distinguish credited balances from pending deposits and pending withdrawals
|
||||
- publish current internal inventory state
|
||||
- expose freshness and reconciliation state
|
||||
|
||||
Inputs:
|
||||
- NEAR Intents inventory or verifier surfaces
|
||||
- liquidity-manager state when relevant
|
||||
|
||||
Outputs:
|
||||
- `state.intent_inventory`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /refresh`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
|
||||
Required state:
|
||||
- spendable balances by asset
|
||||
- pending inbound funding by asset
|
||||
- pending outbound withdrawal by asset
|
||||
- last sync time
|
||||
- reconciliation status
|
||||
|
||||
Important edge cases:
|
||||
- internal inventory surface unavailable
|
||||
- external deposit seen but not yet credited internally
|
||||
- credited balance lower than expected after funding
|
||||
- stale inventory snapshot
|
||||
|
||||
Required behavior:
|
||||
- only credited internal inventory counts as spendable
|
||||
- pending treasury movements must remain non-spendable
|
||||
- stale inventory state must be visible and actionable
|
||||
|
||||
### 4. `liquidity-manager`
|
||||
Responsibilities:
|
||||
- request and track deposit addresses or equivalent funding handles for treasury assets
|
||||
- track treasury funding actions from external wallets into NEAR Intents
|
||||
- track withdrawals and rebalance actions
|
||||
- expose current funding pipeline state
|
||||
- publish auditable liquidity action records
|
||||
|
||||
Inputs:
|
||||
- treasury configuration
|
||||
- external funding wallets
|
||||
- NEAR Intents deposit or withdrawal surfaces
|
||||
|
||||
Outputs:
|
||||
- `ops.liquidity_action`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /refresh`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
- `POST /freeze-withdrawals`
|
||||
|
||||
Required state:
|
||||
- active deposit addresses or funding handles
|
||||
- recent funding attempts
|
||||
- pending credits
|
||||
- recent withdrawals
|
||||
- rebalance state
|
||||
|
||||
Important edge cases:
|
||||
- deposit address request failure
|
||||
- external wallet funded but NEAR Intents credit delayed
|
||||
- duplicate funding detection
|
||||
- unsupported asset or chain mapping
|
||||
|
||||
Required behavior:
|
||||
- treasury actions must remain visible and auditable
|
||||
- funding must be decoupled from the per-trade hot path
|
||||
|
||||
### 5. `history-writer`
|
||||
Responsibilities:
|
||||
- consume the core topics
|
||||
- write append-only rows into PostgreSQL
|
||||
- preserve causality across quote, decision, and execution records
|
||||
- expose lag and write state
|
||||
|
||||
Inputs:
|
||||
- all core Kafka topics
|
||||
|
||||
Outputs:
|
||||
- PostgreSQL rows
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
- `POST /drain`
|
||||
|
||||
Required stored record families:
|
||||
- raw quotes
|
||||
- normalized demand
|
||||
- reference prices
|
||||
- inventory snapshots
|
||||
- liquidity actions
|
||||
- trade decisions
|
||||
- execute commands
|
||||
- execution results
|
||||
|
||||
Important edge cases:
|
||||
- PostgreSQL unavailable
|
||||
- duplicate deliveries from Kafka
|
||||
- offset commit mismatch after partial write
|
||||
|
||||
Required behavior:
|
||||
- never claim success before the write is durable
|
||||
- surface last committed offsets and last successful write time
|
||||
|
||||
### 6. `strategy-engine`
|
||||
Responsibilities:
|
||||
- join latest demand with latest price and inventory state
|
||||
- compute implied quote rate
|
||||
- compare it to fair rate
|
||||
- apply freshness thresholds
|
||||
- apply the initial 2% gross edge threshold
|
||||
- apply max-notional and inventory checks
|
||||
- emit auditable decision events
|
||||
- emit `cmd.execute_trade` only when all gates pass and the system is armed
|
||||
|
||||
Inputs:
|
||||
- `norm.swap_demand`
|
||||
- `ref.market_price`
|
||||
- `state.intent_inventory`
|
||||
|
||||
Outputs:
|
||||
- `decision.trade_decision`
|
||||
- `cmd.execute_trade`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /arm`
|
||||
- `POST /disarm`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
- `PUT /threshold`
|
||||
- `PUT /limits`
|
||||
|
||||
Required decision state:
|
||||
- arm state
|
||||
- current threshold
|
||||
- current notional cap
|
||||
- latest decision
|
||||
- latest rejected decision reason
|
||||
- per-reason skip counters
|
||||
|
||||
Required decision record fields:
|
||||
- `decision_id`
|
||||
- `quote_id`
|
||||
- `pair`
|
||||
- `direction`
|
||||
- `implied_rate`
|
||||
- `reference_rate`
|
||||
- `gross_edge_pct`
|
||||
- `price_freshness_ms`
|
||||
- `inventory_snapshot`
|
||||
- `decision`
|
||||
- `decision_reason`
|
||||
|
||||
Important edge cases:
|
||||
- stale prices
|
||||
- no price yet
|
||||
- no inventory yet
|
||||
- insufficient spendable inventory
|
||||
- pending deposit exists but is not yet credited
|
||||
- quote already expired
|
||||
- repeated quote IDs
|
||||
- one direction affordable and the other not
|
||||
- fresh deploy starts armed by mistake
|
||||
|
||||
### 7. `trade-executor`
|
||||
Responsibilities:
|
||||
- consume `cmd.execute_trade`
|
||||
- load the correct Near Intents signing authority
|
||||
- perform the real Near Intents submission using pre-funded internal inventory
|
||||
- preserve idempotency across retries and restarts
|
||||
- emit `exec.trade_result`
|
||||
|
||||
Inputs:
|
||||
- `cmd.execute_trade`
|
||||
- signing secrets
|
||||
- optional latest inventory state
|
||||
|
||||
Outputs:
|
||||
- `exec.trade_result`
|
||||
|
||||
Control surface:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
- `POST /arm`
|
||||
- `POST /disarm`
|
||||
- `POST /pause`
|
||||
- `POST /resume`
|
||||
- `POST /drain`
|
||||
|
||||
Required state:
|
||||
- arm state
|
||||
- last command seen
|
||||
- last request sent
|
||||
- last venue response
|
||||
- in-flight command count
|
||||
- completed command count
|
||||
- error counters
|
||||
|
||||
Important edge cases:
|
||||
- duplicate command delivery
|
||||
- crash after submission but before result publish
|
||||
- venue reject
|
||||
- timeout with unknown outcome
|
||||
- insufficient internal inventory detected late
|
||||
- secret missing or invalid
|
||||
- signer not authorized for the intended NEAR Intents account
|
||||
|
||||
Required behavior:
|
||||
- never silently swallow a venue error
|
||||
- always publish a result record for attempted commands
|
||||
- make duplicate suppression durable
|
||||
- never try to bridge or top up inventory during trade execution
|
||||
- start disarmed by default on fresh deploy
|
||||
|
||||
## Persistence design
|
||||
|
||||
### Why PostgreSQL now
|
||||
- current scale does not require a special time-series database
|
||||
- rows are easier to inspect than a custom file format for this phase
|
||||
- joins across decision, command, and result matter more right now than raw ingestion throughput
|
||||
- treasury, inventory, and execution joins matter more than raw bus throughput at this stage
|
||||
|
||||
### Minimum tables or equivalent record families
|
||||
- `raw_near_intents_quotes`
|
||||
- `swap_demand_events`
|
||||
- `market_price_events`
|
||||
- `intent_inventory_snapshots`
|
||||
- `liquidity_actions`
|
||||
- `trade_decisions`
|
||||
- `execute_trade_commands`
|
||||
- `trade_execution_results`
|
||||
|
||||
### Query requirements
|
||||
Must be able to answer:
|
||||
- what was the latest fair price when this decision was made
|
||||
- what spendable inventory existed when this decision was made
|
||||
- what treasury funding actions were still pending
|
||||
- why was this quote skipped
|
||||
- what command was emitted for this quote
|
||||
- what happened when the executor submitted it
|
||||
- what credited internal inventory existed at the time
|
||||
|
||||
## Control and stop semantics
|
||||
Every service must support inspection.
|
||||
|
||||
State endpoints must be sufficient to answer:
|
||||
- is it healthy
|
||||
- is it connected
|
||||
- what is it currently using as input state
|
||||
- what was the last successful action
|
||||
- why is it blocked, if blocked
|
||||
- whether the state is authoritative or stale
|
||||
|
||||
Stopping must be explicit:
|
||||
- `pause` means stop taking new work but keep process alive
|
||||
- `drain` means finish in-flight work then stop cleanly
|
||||
- `disarm` means remain live and observable but refuse side effects
|
||||
|
||||
This matters because the operator must be able to halt strategy or execution without destroying the whole pipeline.
|
||||
|
||||
## Logging requirements
|
||||
All services use structured JSON logs.
|
||||
|
||||
Stable top-level fields:
|
||||
- `level`
|
||||
- `service`
|
||||
- `component`
|
||||
- `event`
|
||||
- `namespace`
|
||||
- `venue`
|
||||
- `topic`
|
||||
- `pair`
|
||||
|
||||
Additional body fields when relevant:
|
||||
- `quote_id`
|
||||
- `decision_id`
|
||||
- `command_id`
|
||||
- `execution_id`
|
||||
- `inventory_id`
|
||||
- `liquidity_action_id`
|
||||
- `gross_edge_pct`
|
||||
- `price_freshness_ms`
|
||||
|
||||
Log when:
|
||||
- source connection is lost or reestablished
|
||||
- price source becomes stale
|
||||
- inventory state becomes stale or recovers
|
||||
- funding action is requested, seen, credited, delayed, failed, or frozen
|
||||
- strategy rejects with a meaningful reason
|
||||
- strategy arms or disarms
|
||||
- executor arms or disarms
|
||||
- command is submitted
|
||||
- venue rejects or times out
|
||||
- PostgreSQL disconnects or recovers
|
||||
- control API changes state
|
||||
|
||||
Do not log every healthy message by default.
|
||||
|
||||
## Failure and failover behavior
|
||||
|
||||
### Reference pricing
|
||||
- Kraken failure with fresh CoinGecko:
|
||||
- allowed only if fallback policy says yes
|
||||
- decision records must note fallback source use
|
||||
- both sources stale:
|
||||
- strategy blocks all execution
|
||||
|
||||
### Inventory state
|
||||
- missing or stale inventory state:
|
||||
- strategy may still emit a non-actionable rejected decision
|
||||
- strategy must not emit `cmd.execute_trade`
|
||||
- pending funding action:
|
||||
- remains non-spendable
|
||||
- must be visible in control state and durable records
|
||||
- treasury action service down:
|
||||
- already funded trading may continue if inventory-sync is fresh
|
||||
- new funding or withdrawal operations must be blocked visibly
|
||||
|
||||
### Persistence
|
||||
- PostgreSQL unavailable:
|
||||
- history-writer unhealthy
|
||||
- system must surface that audit history is impaired
|
||||
- if the user wants strict mode, strategy or executor may be blocked until persistence returns
|
||||
|
||||
### Execution
|
||||
- timeout with unknown venue outcome:
|
||||
- emit explicit uncertain result
|
||||
- preserve idempotency state for recovery
|
||||
- restart after partial submission:
|
||||
- executor must not blindly resubmit without checking prior state
|
||||
- executor sees lower real inventory than strategy snapshot:
|
||||
- emit explicit failure result
|
||||
- do not attempt fallback treasury movement
|
||||
|
||||
## Testing and validation plan
|
||||
|
||||
### Unit tests
|
||||
- implied-rate calculation for both directions
|
||||
- explicit EURe/EUR pricing-basis test
|
||||
- threshold checks
|
||||
- stale-price blocking
|
||||
- insufficient-inventory blocking
|
||||
- pending-deposit-not-spendable blocking
|
||||
- command emission only when armed
|
||||
- idempotency transitions in executor
|
||||
|
||||
### Integration tests
|
||||
- pricing service emits canonical reference-price events
|
||||
- inventory-sync emits credited vs pending inventory state correctly
|
||||
- liquidity-manager records funding actions and status transitions
|
||||
- strategy consumes demand, pricing, and inventory and emits decision plus command as expected
|
||||
- history-writer persists linked records across topics
|
||||
- executor publishes result records for success and failure paths
|
||||
|
||||
### Runtime validation in cluster
|
||||
- inspect each service `/state`
|
||||
- verify live reference pricing updates
|
||||
- verify live internal inventory state
|
||||
- verify one treasury funding path from request or deposit tracking to credited inventory
|
||||
- verify strategy and executor start disarmed on a fresh deploy
|
||||
- observe a rejected decision due to block condition
|
||||
- arm strategy and executor
|
||||
- keep the first live max notional tiny, on the order of a few EURe, before any larger cap
|
||||
- observe one command emission
|
||||
- observe one real Near Intents execution attempt
|
||||
- verify PostgreSQL contains the full linked chain
|
||||
|
||||
## Deliberately rejected for this turn
|
||||
- dashboards
|
||||
- broad multi-pair abstractions
|
||||
- ML training infrastructure
|
||||
- broad backtest framework
|
||||
- polished operator UI
|
||||
- warehouse or lakehouse design
|
||||
|
||||
## Expected deliverables
|
||||
- `market-reference-ingest`
|
||||
- `inventory-sync`
|
||||
- `liquidity-manager`
|
||||
- `history-writer`
|
||||
- `strategy-engine`
|
||||
- `trade-executor`
|
||||
- PostgreSQL schema and migrations or equivalent setup
|
||||
- updated Kafka topic creation and config
|
||||
- shared control API pattern for all long-running services
|
||||
- tests for strategy, inventory, persistence, and executor behavior
|
||||
- docs for operations, arm or disarm flow, and inspection commands
|
||||
|
||||
## Validation target
|
||||
This turn is only complete when the deployed system can, through repo-controlled services alone, take one live active-pair quote, price it, verify credited internal inventory, decide on it, gate it, submit a real Near Intents execution attempt, and preserve the full record chain in PostgreSQL.
|
||||
|
|
@ -0,0 +1,356 @@
|
|||
# Implementation Proof: first non-mocked tradeable loop for one pair
|
||||
|
||||
Status: open
|
||||
Opened: 2026-04-01
|
||||
|
||||
## Target outcome
|
||||
The active turn is not complete when the repo merely observes demand or emits placeholder commands. It is complete only when the deployed system can make a real Near Intents trade attempt for the active BTC/Gnosis EURe pair using pre-funded inventory already credited inside NEAR Intents, with the full chain stored durably for later analytics and backtesting.
|
||||
|
||||
## Hypothesis
|
||||
`unrip` becomes materially more real once one live NEAR swap-demand event can move through this full chain without mocks:
|
||||
|
||||
1. real quote and demand ingestion
|
||||
2. live external reference pricing
|
||||
3. synchronized spendable inventory state from NEAR Intents
|
||||
4. auditable strategy decision
|
||||
5. inventory and safety gating
|
||||
6. real Near Intents execution attempt using credited internal inventory
|
||||
7. durable storage of the full event chain
|
||||
|
||||
If any of those links is still dummy, manual-only, or opaque after the fact, the proof is not achieved.
|
||||
|
||||
## Active pair and trading rule
|
||||
- Active pair:
|
||||
- `nep141:btc.omft.near`
|
||||
- `nep141:gnosis-0x420ca0f9b9b604ce0fd9c18ef134c705e5fa3430.omft.near`
|
||||
- Both directions must be implemented in the same decision and execution pipeline:
|
||||
- BTC -> EURe
|
||||
- EURe -> BTC
|
||||
- Runtime spendable inventory inside NEAR Intents decides which direction can actually fire.
|
||||
- External Gnosis and BTC wallets are treasury and funding inputs, not the per-trade spend path.
|
||||
- Pending deposits or withdrawals must not be treated as spendable inventory.
|
||||
- Initial decision rule:
|
||||
- use Kraken plus CoinGecko reference prices
|
||||
- make the EURe/EUR pricing assumption explicit in code and decision records
|
||||
- require a gross edge of at least 2%
|
||||
- require fresh reference pricing
|
||||
- require fresh internal inventory state
|
||||
- require sufficient spendable source-asset inventory already credited inside NEAR Intents
|
||||
- keep strategy and executor disarmed by default on deploy
|
||||
- start with a very small max notional for the first live attempt
|
||||
|
||||
## Required services
|
||||
|
||||
### `near-intents-ingest`
|
||||
Responsibilities:
|
||||
- connect to the live NEAR Intents quote feed
|
||||
- maintain the active pair filter
|
||||
- publish raw venue quotes
|
||||
- publish normalized swap-demand events
|
||||
- expose current ingest state
|
||||
|
||||
Must not:
|
||||
- make trading decisions
|
||||
- execute trades
|
||||
|
||||
### `market-reference-ingest`
|
||||
Responsibilities:
|
||||
- maintain current reference pricing for BTC/EUR and mapped BTC/EURe fair value
|
||||
- use Kraken as the primary fast path
|
||||
- use CoinGecko as fallback or cross-check
|
||||
- publish reference-price events
|
||||
- expose latest price state, freshness, and source health
|
||||
|
||||
Must not:
|
||||
- emit trade commands
|
||||
|
||||
### `inventory-sync`
|
||||
Responsibilities:
|
||||
- read current spendable inventory from the NEAR Intents internal ledger or equivalent supported inventory surface
|
||||
- distinguish credited inventory from pending deposits and withdrawals
|
||||
- publish or serve current internal inventory state
|
||||
- expose freshness, last successful sync time, and reconciliation status
|
||||
|
||||
Must not:
|
||||
- initiate treasury movements
|
||||
- make trading decisions
|
||||
|
||||
### `liquidity-manager`
|
||||
Responsibilities:
|
||||
- request or manage deposit addresses for supported treasury assets and chains
|
||||
- track funding actions from external treasury wallets into NEAR Intents
|
||||
- track pending deposits, credited deposits, withdrawals, and rebalance actions
|
||||
- expose the current funding and treasury state
|
||||
- publish or record liquidity actions so they are auditable
|
||||
|
||||
Must not:
|
||||
- execute trading strategy
|
||||
- spend inventory on the trade hot path
|
||||
|
||||
### `history-writer`
|
||||
Responsibilities:
|
||||
- consume the core Kafka topics
|
||||
- write append-only records into PostgreSQL
|
||||
- preserve enough IDs and timestamps to reconstruct causality
|
||||
- expose writer lag, last committed offsets, and write health
|
||||
|
||||
Must not:
|
||||
- serve as the message bus
|
||||
- make trading decisions
|
||||
|
||||
### `strategy-engine`
|
||||
Responsibilities:
|
||||
- consume normalized demand plus fresh reference prices
|
||||
- consume fresh spendable inventory state
|
||||
- compute implied rate from the quote
|
||||
- compare quote rate against external reference pricing
|
||||
- apply the initial 2% gross edge threshold
|
||||
- apply freshness, arming, and notional checks
|
||||
- apply inventory-aware direction gating
|
||||
- emit auditable decision events
|
||||
- emit `cmd.execute_trade` only when a decision is actionable
|
||||
|
||||
Must not:
|
||||
- hold private keys
|
||||
- execute venue actions directly
|
||||
|
||||
### `trade-executor`
|
||||
Responsibilities:
|
||||
- consume `cmd.execute_trade`
|
||||
- load the correct Near Intents signing authority at runtime
|
||||
- perform the actual Near Intents submission against pre-funded internal inventory
|
||||
- preserve idempotency and duplicate suppression
|
||||
- publish `exec.trade_result`
|
||||
- expose current arm state, recent attempts, and last venue error
|
||||
|
||||
Must not:
|
||||
- invent trading logic
|
||||
- bypass Kafka and manual safety gates
|
||||
- perform ad hoc bridging or treasury funding on the trade hot path
|
||||
|
||||
## Required durable storage
|
||||
PostgreSQL is the first durable analytics and audit store.
|
||||
|
||||
It must store at least:
|
||||
- raw quotes
|
||||
- normalized demand
|
||||
- reference-price snapshots
|
||||
- internal inventory snapshots
|
||||
- liquidity and funding actions
|
||||
- strategy decisions
|
||||
- trade commands
|
||||
- execution results
|
||||
- treasury status and reconciliation metadata
|
||||
|
||||
Why PostgreSQL first:
|
||||
- fast enough for the current streaming volume
|
||||
- queryable for inspection and analytics
|
||||
- simple to operate in the current cluster
|
||||
- good fit for append-only audit plus current-state views
|
||||
|
||||
Kafka remains the streaming backbone. PostgreSQL is not the hot transport path.
|
||||
|
||||
## Required control surface
|
||||
Every long-running service in this turn must expose a small HTTP control surface. The point is not a polished UI. The point is operability.
|
||||
|
||||
At minimum every service must expose:
|
||||
- `GET /healthz`
|
||||
- `GET /state`
|
||||
|
||||
Services with runtime controls must also expose the relevant action endpoints:
|
||||
|
||||
### `near-intents-ingest`
|
||||
- inspect:
|
||||
- pair filter
|
||||
- connection state
|
||||
- frames received
|
||||
- published counts
|
||||
- control:
|
||||
- update pair filter
|
||||
- disable or reset pair filter
|
||||
|
||||
### `market-reference-ingest`
|
||||
- inspect:
|
||||
- latest Kraken price
|
||||
- latest CoinGecko price
|
||||
- derived fair price
|
||||
- freshness age
|
||||
- source health
|
||||
- control:
|
||||
- pause or resume polling or streaming
|
||||
- trigger ad hoc refresh
|
||||
|
||||
### `inventory-sync`
|
||||
- inspect:
|
||||
- latest spendable balances by asset
|
||||
- pending deposits and withdrawals
|
||||
- last sync time
|
||||
- reconciliation status
|
||||
- control:
|
||||
- trigger sync
|
||||
- pause or resume sync
|
||||
|
||||
### `liquidity-manager`
|
||||
- inspect:
|
||||
- active deposit addresses
|
||||
- recent funding actions
|
||||
- pending deposits
|
||||
- credited deposits
|
||||
- recent withdrawals
|
||||
- control:
|
||||
- request or rotate deposit address where supported
|
||||
- refresh treasury status
|
||||
- pause or resume funding trackers
|
||||
- disable withdrawals or rebalance actions
|
||||
|
||||
### `history-writer`
|
||||
- inspect:
|
||||
- last persisted offsets by topic
|
||||
- last write time
|
||||
- error count
|
||||
- database connectivity
|
||||
- control:
|
||||
- pause writes
|
||||
- resume writes
|
||||
- drain and stop cleanly
|
||||
|
||||
### `strategy-engine`
|
||||
- inspect:
|
||||
- arm state
|
||||
- active threshold
|
||||
- latest reference snapshot used
|
||||
- latest inventory snapshot used
|
||||
- latest decisions and reasons
|
||||
- skipped counts by reason
|
||||
- control:
|
||||
- arm or disarm
|
||||
- pause or resume decisions
|
||||
- change threshold
|
||||
- set notional cap
|
||||
|
||||
### `trade-executor`
|
||||
- inspect:
|
||||
- arm state
|
||||
- last command received
|
||||
- last venue response
|
||||
- last error
|
||||
- in-flight and completed command counts
|
||||
- control:
|
||||
- arm or disarm execution
|
||||
- pause consumption
|
||||
- drain and stop cleanly
|
||||
|
||||
Stopping a service must mean a graceful stop or drain, not “kill the pod and hope.”
|
||||
|
||||
## Logging and observability requirements
|
||||
All services must emit structured JSON logs with stable fields.
|
||||
|
||||
Stable fields:
|
||||
- `level`
|
||||
- `service`
|
||||
- `component`
|
||||
- `event`
|
||||
- `namespace`
|
||||
- `venue`
|
||||
- `topic`
|
||||
- `pair`
|
||||
|
||||
High-cardinality IDs belong in the body, not labels:
|
||||
- `quote_id`
|
||||
- `command_id`
|
||||
- `decision_id`
|
||||
- `execution_id`
|
||||
|
||||
What must be logged:
|
||||
- connection loss and recovery
|
||||
- source stale state
|
||||
- inventory-sync stale state
|
||||
- treasury funding action requested, credited, failed, or stuck
|
||||
- invalid messages
|
||||
- decision accepted and decision rejected with reason
|
||||
- arm and disarm actions
|
||||
- execution submitted, rejected, failed, recovered, completed
|
||||
- PostgreSQL disconnect and recovery
|
||||
- control API failures
|
||||
|
||||
What should not be logged:
|
||||
- per-message noise without state change
|
||||
- full payload spam by default
|
||||
|
||||
## Required edge cases and failure handling
|
||||
- stale reference prices must block decisions
|
||||
- stale or missing internal inventory state must block execution
|
||||
- insufficient credited inventory must block the affected direction only
|
||||
- pending deposits must not be counted as spendable inventory
|
||||
- deposit address created but never funded must not be mistaken for liquidity
|
||||
- external transfer observed but not yet credited in NEAR Intents must stay non-spendable
|
||||
- credited inventory drift between inventory-sync and executor must hard-fail the command and publish a result
|
||||
- Kraken down but CoinGecko healthy should degrade, not crash, if policy allows fallback
|
||||
- both reference sources stale must block decisions
|
||||
- PostgreSQL down must surface a hard health failure and stop claiming the system is trade-ready
|
||||
- liquidity-manager failure must block new funding operations but must not silently stop already funded trading
|
||||
- duplicate `cmd.execute_trade` must not cause duplicate venue submission
|
||||
- executor restart after a partial submission must preserve idempotency behavior
|
||||
- Near Intents API or RPC failures must become explicit `trade_result` failure records
|
||||
- pair inactivity must not be mistaken for pipeline breakage
|
||||
|
||||
## Definition of done
|
||||
- The deployed cluster is running all required services for this loop.
|
||||
- Live Kraken and CoinGecko reference prices are flowing and inspectable.
|
||||
- Spendable inventory inside NEAR Intents is flowing and inspectable.
|
||||
- At least one treasury funding path is documented and verified:
|
||||
- deposit address or supported funding path created
|
||||
- funding action observed
|
||||
- credited inventory visible in internal state
|
||||
- PostgreSQL contains the full event chain for at least one real quote path.
|
||||
- Strategy decisions are non-dummy and include explicit reason fields with edge, freshness, and inventory context.
|
||||
- Both trade directions are implemented in the same decision and execution path.
|
||||
- At least one direction can be armed based on available credited inventory.
|
||||
- A real Near Intents execution attempt can be triggered through the repo-controlled path.
|
||||
- The resulting venue response is captured durably and inspectably.
|
||||
- Each service exposes the required health and state endpoints, and controlled pause or arm semantics where appropriate.
|
||||
- Fresh deploys start disarmed by default and require explicit operator arming before side effects.
|
||||
- Validation includes not only happy path but blocked-path evidence:
|
||||
- stale reference blocked
|
||||
- insufficient inventory blocked
|
||||
- pending-deposit-not-spendable blocked
|
||||
- disarmed executor blocked
|
||||
|
||||
## Current real
|
||||
- Real NEAR Intents quote data is flowing into the cluster.
|
||||
- The pair filter can be configured and changed at runtime.
|
||||
- Raw and normalized events already exist in Redpanda topics.
|
||||
- The app is deployable and observable in Kubernetes.
|
||||
- Kafka already provides the event backbone.
|
||||
|
||||
## Current fake or incomplete
|
||||
- Reference pricing does not exist yet.
|
||||
- Spendable internal inventory sync does not exist yet.
|
||||
- Treasury funding or deposit management does not exist yet.
|
||||
- Strategy decisions are still dummy placeholders.
|
||||
- Execution is still dummy.
|
||||
- The current executor is not a real Near Intents adapter.
|
||||
- PostgreSQL is not yet part of the loop.
|
||||
- Most services do not yet have their own control surfaces.
|
||||
- Real signing and treasury secret handling is not yet wired.
|
||||
|
||||
## Failure conditions
|
||||
This proof fails if any of the following is true:
|
||||
- a trade can only happen via manual shell steps or out-of-band operator intervention
|
||||
- the system depends on ad hoc bridging or external-wallet spending on the trade hot path
|
||||
- decision logic is still embedded in a dummy or placeholder service
|
||||
- executor logic is still simulated
|
||||
- a service cannot be inspected or paused cleanly at runtime
|
||||
- the system cannot explain why a quote did or did not become a trade
|
||||
- the stored history cannot reconstruct quote, pricing, decision, command, and result
|
||||
- both directions are not implemented
|
||||
- the system attempts to execute without credited internal source inventory
|
||||
|
||||
## Expected validation evidence
|
||||
- live `ref.market_price` events and current state from the pricing control API
|
||||
- live internal inventory state from the inventory-sync control API
|
||||
- PostgreSQL rows linking quote, reference, inventory snapshot, decision, command, and execution result
|
||||
- a blocked decision caused by stale price or insufficient inventory
|
||||
- a funding action that becomes credited inventory
|
||||
- a successful command emission from the strategy service with a recorded edge above 2%
|
||||
- one real Near Intents execution attempt with stored request and response metadata
|
||||
1
archive/research/.gitkeep
Normal file
1
archive/research/.gitkeep
Normal file
|
|
@ -0,0 +1 @@
|
|||
|
||||
14
research/ACTIVE.md
Normal file
14
research/ACTIVE.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# Research Turn
|
||||
|
||||
Status: idle
|
||||
|
||||
No approved research turn is active yet.
|
||||
|
||||
When opening one, capture:
|
||||
- charter
|
||||
- hypothesis
|
||||
- dataset or source of truth
|
||||
- metrics
|
||||
- assumptions
|
||||
- falsification condition
|
||||
- expected artifact paths under `research/experiments/`
|
||||
9
research/QUEUE.md
Normal file
9
research/QUEUE.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Research Queue
|
||||
|
||||
This queue holds strategy and analysis questions that are not yet an active research turn.
|
||||
|
||||
## Candidate charters
|
||||
- Determine which Kraken symbols and CoinGecko asset IDs give a trustworthy pricing basis for the active pair.
|
||||
- Measure how often the active pair's implied rate diverges from external reference prices after a simple fee model.
|
||||
- Test whether stale quotes correlate with worse downstream execution quality or higher reject rates.
|
||||
- Import the local historical EUR/BTC data and decide how it should seed replay and backtesting.
|
||||
43
scripts/workflow/add_backlog.py
Executable file
43
scripts/workflow/add_backlog.py
Executable file
|
|
@ -0,0 +1,43 @@
|
|||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from common import BACKLOG_PATH, BACKLOG_SECTION, insert_into_section, load_text, next_backlog_id, save_text
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Append an item to BACKLOG.md.")
|
||||
parser.add_argument(
|
||||
"--lane",
|
||||
required=True,
|
||||
choices=("implementation", "research", "ops", "bug"),
|
||||
help="Backlog section to append to.",
|
||||
)
|
||||
parser.add_argument("--summary", required=True, help="One-line backlog summary.")
|
||||
parser.add_argument(
|
||||
"--priority",
|
||||
default="soon",
|
||||
choices=("now", "soon", "later"),
|
||||
help="Coarse urgency label.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tags",
|
||||
default="",
|
||||
help="Comma-separated tags stored inline for grepability.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
item_id = next_backlog_id(args.lane)
|
||||
tags = f" tags={args.tags}" if args.tags else ""
|
||||
entry = f"- [{item_id}] ({args.priority}) {args.summary}{tags}"
|
||||
updated = insert_into_section(load_text(BACKLOG_PATH), BACKLOG_SECTION[args.lane], entry)
|
||||
save_text(BACKLOG_PATH, updated)
|
||||
print(item_id)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
150
scripts/workflow/close_turn.py
Executable file
150
scripts/workflow/close_turn.py
Executable file
|
|
@ -0,0 +1,150 @@
|
|||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from common import (
|
||||
ARCHIVE_PATH,
|
||||
IMPLEMENTATION_ARCHIVE_DIR,
|
||||
IMPLEMENTATION_PATH,
|
||||
PROOF_PATH,
|
||||
RESEARCH_ACTIVE_PATH,
|
||||
RESEARCH_ARCHIVE_DIR,
|
||||
append_archive_line,
|
||||
git_commit,
|
||||
load_text,
|
||||
save_text,
|
||||
slugify,
|
||||
timestamp_slug,
|
||||
today_iso,
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Archive and close the current implementation or research turn.")
|
||||
parser.add_argument("--lane", required=True, choices=("implementation", "research"))
|
||||
parser.add_argument(
|
||||
"--status",
|
||||
required=True,
|
||||
choices=("passed", "failed", "paused", "abandoned"),
|
||||
help="Outcome of the turn.",
|
||||
)
|
||||
parser.add_argument("--summary", required=True, help="One-line closure summary.")
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Commit the archive change automatically.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def title_from_content(content: str, prefix: str) -> str:
|
||||
match = re.search(rf"^# {re.escape(prefix)}: (.+)$", content, flags=re.MULTILINE)
|
||||
if not match:
|
||||
raise SystemExit(f"could not find title for {prefix.lower()}")
|
||||
return match.group(1).strip()
|
||||
|
||||
|
||||
def close_implementation_turn(status: str, summary: str) -> tuple[str, list[Path]]:
|
||||
proof_content = load_text(PROOF_PATH)
|
||||
implementation_content = load_text(IMPLEMENTATION_PATH)
|
||||
title = title_from_content(proof_content, "Implementation Proof")
|
||||
slug = slugify(title)
|
||||
stamp = timestamp_slug()
|
||||
|
||||
IMPLEMENTATION_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
proof_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-proof.md"
|
||||
implementation_archive = IMPLEMENTATION_ARCHIVE_DIR / f"{stamp}-{slug}-implementation.md"
|
||||
shutil.copyfile(PROOF_PATH, proof_archive)
|
||||
shutil.copyfile(IMPLEMENTATION_PATH, implementation_archive)
|
||||
|
||||
save_text(
|
||||
PROOF_PATH,
|
||||
"""# Implementation Proof
|
||||
|
||||
Status: idle
|
||||
|
||||
No approved implementation proof is active yet.
|
||||
""",
|
||||
)
|
||||
save_text(
|
||||
IMPLEMENTATION_PATH,
|
||||
"""# Implementation Turn
|
||||
|
||||
Status: idle
|
||||
|
||||
No approved implementation turn is active yet.
|
||||
""",
|
||||
)
|
||||
|
||||
append_archive_line(
|
||||
"implementation",
|
||||
f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}",
|
||||
)
|
||||
return title, [proof_archive, implementation_archive]
|
||||
|
||||
|
||||
def close_research_turn(status: str, summary: str) -> tuple[str, list[Path]]:
|
||||
research_content = load_text(RESEARCH_ACTIVE_PATH)
|
||||
title = title_from_content(research_content, "Research Turn")
|
||||
slug = slugify(title)
|
||||
stamp = timestamp_slug()
|
||||
|
||||
RESEARCH_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
research_archive = RESEARCH_ARCHIVE_DIR / f"{stamp}-{slug}.md"
|
||||
shutil.copyfile(RESEARCH_ACTIVE_PATH, research_archive)
|
||||
|
||||
save_text(
|
||||
RESEARCH_ACTIVE_PATH,
|
||||
"""# Research Turn
|
||||
|
||||
Status: idle
|
||||
|
||||
No approved research turn is active yet.
|
||||
""",
|
||||
)
|
||||
|
||||
append_archive_line(
|
||||
"research",
|
||||
f"- {today_iso()}: `{slug}` closed with status `{status}`. {summary}",
|
||||
)
|
||||
return title, [research_archive]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if args.lane == "implementation" and "Status: idle" in load_text(PROOF_PATH):
|
||||
raise SystemExit("no active implementation turn to close")
|
||||
if args.lane == "research" and "Status: idle" in load_text(RESEARCH_ACTIVE_PATH):
|
||||
raise SystemExit("no active research turn to close")
|
||||
|
||||
if args.lane == "implementation":
|
||||
title, archived_paths = close_implementation_turn(args.status, args.summary)
|
||||
else:
|
||||
title, archived_paths = close_research_turn(args.status, args.summary)
|
||||
|
||||
if args.commit:
|
||||
paths = [ARCHIVE_PATH]
|
||||
if args.lane == "implementation":
|
||||
paths.extend([PROOF_PATH, IMPLEMENTATION_PATH])
|
||||
else:
|
||||
paths.append(RESEARCH_ACTIVE_PATH)
|
||||
paths.extend(archived_paths)
|
||||
git_commit(
|
||||
f"""Archive {args.lane} turn: {title}
|
||||
|
||||
Proof: Preserve the completed {args.lane} turn and record its outcome in the tracked archive.
|
||||
Assumptions: The archived files capture the relevant planning state for the completed turn.
|
||||
Still fake: Archiving does not validate the work by itself; external evidence still governs whether the result is trustworthy.""",
|
||||
paths=paths,
|
||||
)
|
||||
|
||||
for archived_path in archived_paths:
|
||||
print(archived_path.relative_to(RESEARCH_ACTIVE_PATH.parent.parent))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
154
scripts/workflow/common.py
Executable file
154
scripts/workflow/common.py
Executable file
|
|
@ -0,0 +1,154 @@
|
|||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
ROOT_DIR = Path(__file__).resolve().parents[2]
|
||||
BACKLOG_PATH = ROOT_DIR / "BACKLOG.md"
|
||||
ARCHIVE_PATH = ROOT_DIR / "ARCHIVE.md"
|
||||
PROOF_PATH = ROOT_DIR / "PROOF.md"
|
||||
IMPLEMENTATION_PATH = ROOT_DIR / "IMPLEMENTATION.md"
|
||||
RESEARCH_ACTIVE_PATH = ROOT_DIR / "research/ACTIVE.md"
|
||||
IMPLEMENTATION_ARCHIVE_DIR = ROOT_DIR / "archive/implementation"
|
||||
RESEARCH_ARCHIVE_DIR = ROOT_DIR / "archive/research"
|
||||
|
||||
LANE_PREFIX = {
|
||||
"implementation": "I",
|
||||
"research": "R",
|
||||
"ops": "O",
|
||||
"bug": "B",
|
||||
}
|
||||
|
||||
BACKLOG_SECTION = {
|
||||
"implementation": "## Implementation Candidates",
|
||||
"research": "## Research Candidates",
|
||||
"ops": "## Ops Candidates",
|
||||
"bug": "## Bugs",
|
||||
}
|
||||
|
||||
ARCHIVE_SECTION = {
|
||||
"implementation": "## Implementation Turns",
|
||||
"research": "## Research Turns",
|
||||
"planning": "## Planning Events",
|
||||
}
|
||||
|
||||
|
||||
def now_utc() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def today_iso() -> str:
|
||||
return now_utc().date().isoformat()
|
||||
|
||||
|
||||
def timestamp_slug() -> str:
|
||||
return now_utc().strftime("%Y%m%dT%H%M%SZ")
|
||||
|
||||
|
||||
def slugify(value: str) -> str:
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
|
||||
return slug or "turn"
|
||||
|
||||
|
||||
def load_text(path: Path) -> str:
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def save_text(path: Path, content: str) -> None:
|
||||
path.write_text(content.rstrip() + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def insert_into_section(document: str, heading: str, block: str) -> str:
|
||||
lines = document.splitlines()
|
||||
try:
|
||||
start = lines.index(heading)
|
||||
except ValueError as exc:
|
||||
raise SystemExit(f"heading not found: {heading}") from exc
|
||||
|
||||
end = len(lines)
|
||||
for idx in range(start + 1, len(lines)):
|
||||
if lines[idx].startswith("## "):
|
||||
end = idx
|
||||
break
|
||||
|
||||
insert_at = end
|
||||
return "\n".join(lines[:insert_at] + [block] + lines[insert_at:]) + "\n"
|
||||
|
||||
|
||||
def append_archive_line(section: str, line: str) -> None:
|
||||
content = load_text(ARCHIVE_PATH)
|
||||
updated = insert_into_section(content, ARCHIVE_SECTION[section], line)
|
||||
save_text(ARCHIVE_PATH, updated)
|
||||
|
||||
|
||||
def read_backlog_lines() -> list[str]:
|
||||
return load_text(BACKLOG_PATH).splitlines()
|
||||
|
||||
|
||||
def write_backlog_lines(lines: list[str]) -> None:
|
||||
save_text(BACKLOG_PATH, "\n".join(lines))
|
||||
|
||||
|
||||
def next_backlog_id(lane: str) -> str:
|
||||
prefix = LANE_PREFIX[lane]
|
||||
pattern = re.compile(rf"\[{re.escape(prefix)}(\d+)\]")
|
||||
highest = 0
|
||||
for line in read_backlog_lines():
|
||||
match = pattern.search(line)
|
||||
if match:
|
||||
highest = max(highest, int(match.group(1)))
|
||||
return f"{prefix}{highest + 1:03d}"
|
||||
|
||||
|
||||
def backlog_entry_map() -> dict[str, str]:
|
||||
entries: dict[str, str] = {}
|
||||
pattern = re.compile(r"- \[([A-Z]\d+)\] (.+)")
|
||||
for line in read_backlog_lines():
|
||||
match = pattern.match(line)
|
||||
if match:
|
||||
entries[match.group(1)] = match.group(2)
|
||||
return entries
|
||||
|
||||
|
||||
def remove_backlog_ids(ids: list[str]) -> None:
|
||||
id_set = set(ids)
|
||||
pattern = re.compile(r"- \[([A-Z]\d+)\] ")
|
||||
kept: list[str] = []
|
||||
for line in read_backlog_lines():
|
||||
match = pattern.match(line)
|
||||
if match and match.group(1) in id_set:
|
||||
continue
|
||||
kept.append(line)
|
||||
write_backlog_lines(kept)
|
||||
|
||||
|
||||
def git_has_changes() -> bool:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(ROOT_DIR), "status", "--porcelain"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return bool(result.stdout.strip())
|
||||
|
||||
|
||||
def path_has_changes(paths: list[Path]) -> bool:
|
||||
rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths]
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(ROOT_DIR), "status", "--porcelain", "--", *rel_paths],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return bool(result.stdout.strip())
|
||||
|
||||
|
||||
def git_commit(message: str, paths: list[Path]) -> None:
|
||||
if not path_has_changes(paths):
|
||||
return
|
||||
rel_paths = [str(path.relative_to(ROOT_DIR)) for path in paths]
|
||||
subprocess.run(["git", "-C", str(ROOT_DIR), "add", "--", *rel_paths], check=True)
|
||||
subprocess.run(["git", "-C", str(ROOT_DIR), "commit", "-m", message], check=True)
|
||||
9
scripts/workflow/install_hooks.sh
Executable file
9
scripts/workflow/install_hooks.sh
Executable file
|
|
@ -0,0 +1,9 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
|
||||
chmod +x "$ROOT_DIR/.githooks/commit-msg"
|
||||
git -C "$ROOT_DIR" config core.hooksPath .githooks
|
||||
|
||||
echo "Installed tracked git hooks for $ROOT_DIR"
|
||||
176
scripts/workflow/open_turn.py
Executable file
176
scripts/workflow/open_turn.py
Executable file
|
|
@ -0,0 +1,176 @@
|
|||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from common import (
|
||||
BACKLOG_PATH,
|
||||
ARCHIVE_PATH,
|
||||
IMPLEMENTATION_PATH,
|
||||
PROOF_PATH,
|
||||
RESEARCH_ACTIVE_PATH,
|
||||
append_archive_line,
|
||||
backlog_entry_map,
|
||||
git_commit,
|
||||
remove_backlog_ids,
|
||||
save_text,
|
||||
slugify,
|
||||
today_iso,
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Open a new implementation or research turn.")
|
||||
parser.add_argument("--lane", required=True, choices=("implementation", "research"))
|
||||
parser.add_argument("--title", required=True, help="Turn title.")
|
||||
parser.add_argument("--summary", required=True, help="One-line proof or charter summary.")
|
||||
parser.add_argument(
|
||||
"--pick",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Backlog ID to pull into the turn. Repeat as needed.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Commit the planning change automatically.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Replace an already-open turn instead of refusing.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def render_selected(ids: list[str], items: dict[str, str]) -> str:
|
||||
if not ids:
|
||||
return "- none selected"
|
||||
return "\n".join(f"- [{item_id}] {items[item_id]}" for item_id in ids)
|
||||
|
||||
|
||||
def open_implementation_turn(title: str, summary: str, selected: str) -> None:
|
||||
opened = today_iso()
|
||||
save_text(
|
||||
PROOF_PATH,
|
||||
f"""# Implementation Proof: {title}
|
||||
|
||||
Status: open
|
||||
Opened: {opened}
|
||||
|
||||
## Hypothesis
|
||||
{summary}
|
||||
|
||||
## Scope
|
||||
{selected}
|
||||
|
||||
## Non-goals
|
||||
- unchanged from `THESIS.md` unless the user approves otherwise
|
||||
|
||||
## Definition of done
|
||||
- current turn implementation is validated with direct evidence
|
||||
- remaining fakes are listed plainly
|
||||
""",
|
||||
)
|
||||
save_text(
|
||||
IMPLEMENTATION_PATH,
|
||||
f"""# Implementation Turn: {title}
|
||||
|
||||
Status: open
|
||||
Opened: {opened}
|
||||
|
||||
## Goal
|
||||
{summary}
|
||||
|
||||
## Selected backlog items
|
||||
{selected}
|
||||
|
||||
## Notes
|
||||
- Fill in the concrete implementation plan before coding if the live plan no longer matches the turn.
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
def open_research_turn(title: str, summary: str, selected: str) -> None:
|
||||
opened = today_iso()
|
||||
save_text(
|
||||
RESEARCH_ACTIVE_PATH,
|
||||
f"""# Research Turn: {title}
|
||||
|
||||
Status: open
|
||||
Opened: {opened}
|
||||
|
||||
## Charter
|
||||
{summary}
|
||||
|
||||
## Selected backlog items
|
||||
{selected}
|
||||
|
||||
## Hypothesis
|
||||
TBD by the approved research turn.
|
||||
|
||||
## Dataset or source of truth
|
||||
TBD by the approved research turn.
|
||||
|
||||
## Metrics
|
||||
TBD by the approved research turn.
|
||||
|
||||
## Assumptions
|
||||
TBD by the approved research turn.
|
||||
|
||||
## Falsification condition
|
||||
TBD by the approved research turn.
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if args.lane == "implementation":
|
||||
if "Status: open" in PROOF_PATH.read_text(encoding="utf-8") and not args.force:
|
||||
raise SystemExit("implementation turn already open; close it first or pass --force")
|
||||
else:
|
||||
if "Status: open" in RESEARCH_ACTIVE_PATH.read_text(encoding="utf-8") and not args.force:
|
||||
raise SystemExit("research turn already open; close it first or pass --force")
|
||||
|
||||
entries = backlog_entry_map()
|
||||
missing = [item_id for item_id in args.pick if item_id not in entries]
|
||||
if missing:
|
||||
raise SystemExit(f"unknown backlog IDs: {', '.join(missing)}")
|
||||
|
||||
selected = render_selected(args.pick, entries)
|
||||
|
||||
if args.lane == "implementation":
|
||||
open_implementation_turn(args.title, args.summary, selected)
|
||||
else:
|
||||
open_research_turn(args.title, args.summary, selected)
|
||||
|
||||
if args.pick:
|
||||
remove_backlog_ids(args.pick)
|
||||
|
||||
append_archive_line(
|
||||
"planning",
|
||||
(
|
||||
f"- {today_iso()}: opened {args.lane} turn `{slugify(args.title)}` "
|
||||
f"from backlog items {', '.join(args.pick) if args.pick else 'none'}."
|
||||
),
|
||||
)
|
||||
|
||||
if args.commit:
|
||||
commit_paths = [BACKLOG_PATH, ARCHIVE_PATH]
|
||||
if args.lane == "implementation":
|
||||
commit_paths.extend([PROOF_PATH, IMPLEMENTATION_PATH])
|
||||
else:
|
||||
commit_paths.append(RESEARCH_ACTIVE_PATH)
|
||||
git_commit(
|
||||
f"""Open {args.lane} turn: {args.title}
|
||||
|
||||
Proof: Establish the approved {args.lane} turn and move selected backlog items into active scope.
|
||||
Assumptions: The selected backlog items are the approved scope for this turn.
|
||||
Still fake: Opening a turn changes planning state only; the work itself is not implemented yet.""",
|
||||
paths=commit_paths,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
37
scripts/workflow/review_diff.sh
Executable file
37
scripts/workflow/review_diff.sh
Executable file
|
|
@ -0,0 +1,37 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
PROMPT_FILE="$ROOT_DIR/workflow/REVIEW_PROMPT.md"
|
||||
RANGE="${1:-HEAD~1}"
|
||||
|
||||
if [[ "${1:-}" == "--help" ]]; then
|
||||
cat <<'EOF'
|
||||
Usage: bash scripts/workflow/review_diff.sh [git-diff-range]
|
||||
|
||||
Examples:
|
||||
bash scripts/workflow/review_diff.sh HEAD~1
|
||||
bash scripts/workflow/review_diff.sh main...HEAD
|
||||
EOF
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
# Review Bundle
|
||||
|
||||
## Diff range
|
||||
\`$RANGE\`
|
||||
|
||||
## Diff
|
||||
\`\`\`diff
|
||||
EOF
|
||||
|
||||
git -C "$ROOT_DIR" diff "$RANGE"
|
||||
|
||||
cat <<'EOF'
|
||||
```
|
||||
|
||||
## Prompt
|
||||
EOF
|
||||
|
||||
cat "$PROMPT_FILE"
|
||||
28
workflow/REVIEW_PROMPT.md
Normal file
28
workflow/REVIEW_PROMPT.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Adversarial review prompt
|
||||
|
||||
You are reviewing changes for a trading system.
|
||||
|
||||
Your job is to attack the work, not to praise it.
|
||||
|
||||
Look for:
|
||||
|
||||
1. Fake progress.
|
||||
- Did the change make the system more real, or only more elaborate?
|
||||
- Real means contact with live data, validated persistence, verified replay, or explicit evidence.
|
||||
2. Smuggled scope.
|
||||
- Did the diff stay inside the active proof or research charter?
|
||||
3. Unstated assumptions.
|
||||
- Prices, fees, latency, freshness, slippage, API guarantees, retention, and failure modes.
|
||||
4. Placeholder dressed as real.
|
||||
- Hardcoded values, mocks, TODOs, dead branches, or unverifiable claims.
|
||||
5. Missing failure handling.
|
||||
- Focus on real trading-system failures, not abstract defensive-programming trivia.
|
||||
|
||||
For every issue:
|
||||
- state what it is
|
||||
- state why it matters for this system
|
||||
- state whether it should be fixed now, removed, or marked as still fake
|
||||
|
||||
Do not praise style.
|
||||
Do not suggest adjacent features.
|
||||
Do not expand scope.
|
||||
Loading…
Add table
Reference in a new issue