41b9ec680b
Some checks failed
deploy / deploy (push) Failing after 1m35s
Proof: first non-mocked tradeable loop for one pair using funded NEAR Intents inventory, Kafka, and PostgreSQL. Assumptions: solver-side execution is performed by signed token_diff quote responses over the Solver Relay; EURe is treated as 1:1 with EUR; k3s runtime uses unrip-dev.near as the named signer account. Still fake: signer key is not yet registered on intents.near, strategy and executor remain disarmed by default, and no live mainnet quote response has been submitted from this repo yet.
316 lines
11 KiB
Markdown
316 lines
11 KiB
Markdown
# unrip project
|
|
|
|
This repository contains the unrip trading-system code and its project-specific deployment assets.
|
|
|
|
## Contents
|
|
|
|
- `src/` — application code
|
|
- `package.json` / `package-lock.json` — Node package manifest
|
|
- `Dockerfile` / `.dockerignore` — app container build
|
|
- `.env.example` — local app runtime example
|
|
- `compose.yml` — local development stack
|
|
- `deploy/k8s/base/` — project-specific Kubernetes manifests
|
|
- `deploy/redpanda/rpk-topics.txt` — project topic reference
|
|
- `docs/` — project-specific design and contract docs
|
|
|
|
## Local development
|
|
|
|
```bash
|
|
npm install
|
|
cp .env.example .env
|
|
# edit .env
|
|
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Useful commands:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs -f
|
|
docker compose logs -f \
|
|
near-intents-ingest market-reference-ingest liquidity-manager \
|
|
inventory-sync history-writer strategy-engine trade-executor
|
|
npm run near-intents:ingest
|
|
npm run market-reference:ingest
|
|
npm run liquidity:manager
|
|
npm run inventory:sync
|
|
npm run history:writer
|
|
npm run strategy:engine
|
|
npm run trade:executor
|
|
```
|
|
|
|
## App image
|
|
|
|
The app image is now built from this directory.
|
|
|
|
Examples:
|
|
|
|
```bash
|
|
docker build -t unrip:dev .
|
|
```
|
|
|
|
## Kubernetes manifests
|
|
|
|
Project manifests live under:
|
|
|
|
- `deploy/k8s/base/`
|
|
|
|
The shared cluster/platform resources live in the separate infra repository.
|
|
|
|
## Deployment
|
|
|
|
This repo is the app-side deployment repo. The shared Hetzner/k3s bootstrap,
|
|
Forgejo runner, registry, and other platform services live in the separate
|
|
platform repo.
|
|
|
|
See `docs/deployment.md` for the full operator path.
|
|
See `docs/operator-runbook.md` for the live turn control and arming sequence.
|
|
|
|
### One-time app bootstrap
|
|
|
|
Bootstrap the app namespace, secrets, and Forgejo repo settings from this repo:
|
|
|
|
```bash
|
|
bash scripts/deploy/bootstrap.sh
|
|
```
|
|
|
|
That bootstrap also refreshes the local `forgejo` remote URL for HTTPS pushes
|
|
when it has enough auth material to do so.
|
|
|
|
By default, the script uses the adjacent platform checkout at `../unrip3` for:
|
|
- `kubeconfig.yaml`
|
|
- `kubeconfig.incluster.yaml`
|
|
- registry credentials
|
|
- the `NEAR_INTENTS_API_KEY` fallback from `../unrip3/.env`
|
|
|
|
If you are not using that local split, provide the values yourself via env vars
|
|
such as `KUBECONFIG_PATH`, `CI_KUBECONFIG_PATH`, `REGISTRY_HOST`,
|
|
`REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `NEAR_INTENTS_API_KEY`, and either
|
|
`FORGEJO_TOKEN` or `FORGEJO_ADMIN_USERNAME` / `FORGEJO_ADMIN_PASSWORD`.
|
|
|
|
### Routine deploy
|
|
|
|
After bootstrap, deployment is just a push to Forgejo `main`:
|
|
|
|
```bash
|
|
git push forgejo main
|
|
```
|
|
|
|
`.forgejo/workflows/deploy.yml` then:
|
|
- applies `deploy/k8s/base`
|
|
- builds the image from this repo root inside the cluster with Kaniko
|
|
- pushes it to the shared registry
|
|
- rolls the `unrip` deployments
|
|
- uses a fresh temporary runner workspace on each run, so reruns do not require
|
|
manual cleanup on the Forgejo runner
|
|
|
|
### Observe rollout
|
|
|
|
```bash
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip get deploy,pods,job
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/near-intents-ingest
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/market-reference-ingest
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/liquidity-manager
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/inventory-sync
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/history-writer
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/strategy-engine
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/trade-executor
|
|
```
|
|
|
|
### Auxiliary ops scripts
|
|
|
|
These scripts default to the same adjacent platform checkout as the deployment
|
|
bootstrap: `../unrip3/.state/hetzner/kubeconfig.yaml`. Override with
|
|
`KUBECONFIG_PATH`, `KUBECONFIG`, or `PLATFORM_REPO_DIR` if needed.
|
|
|
|
`scripts/ops/deployment_status.py`
|
|
- Shows current deployment readiness, pod uptime, restart counts, and mounted storage usage.
|
|
- Outputs JSON by default so it can be piped directly into `jq`.
|
|
- By default it shows the live deployment pods only.
|
|
- Use `--include-completed` to include completed Job pods.
|
|
- Use `--include-rootfs` if you also want a probe of `/` for pods without PVC-backed mounts.
|
|
- Use `--output table` for the human-readable table view.
|
|
|
|
```bash
|
|
python3 scripts/ops/deployment_status.py
|
|
python3 scripts/ops/deployment_status.py | jq '.pods[] | {name, uptime}'
|
|
python3 scripts/ops/deployment_status.py --include-completed
|
|
python3 scripts/ops/deployment_status.py --output table
|
|
```
|
|
|
|
`scripts/ops/redpanda_storage.py`
|
|
- Shows how much data Redpanda is currently storing for the unrip topics.
|
|
- Outputs JSON by default so it can be piped directly into `jq`.
|
|
- Includes per-topic local bytes, total bytes, segment counts, and the overall Redpanda data-path usage.
|
|
- Use `--all-topics` to inspect every visible topic, or `--topic` multiple times for a subset.
|
|
- Use `--output table` for the human-readable table view.
|
|
|
|
```bash
|
|
python3 scripts/ops/redpanda_storage.py
|
|
python3 scripts/ops/redpanda_storage.py | jq '.totals'
|
|
python3 scripts/ops/redpanda_storage.py --all-topics
|
|
python3 scripts/ops/redpanda_storage.py --topic raw.near_intents.quote --topic norm.swap_demand
|
|
python3 scripts/ops/redpanda_storage.py --output table
|
|
```
|
|
|
|
`scripts/ops/live_near_intents.py`
|
|
- Reads the raw NEAR quote stream entering Redpanda.
|
|
- Outputs clean JSON by default. Bounded reads return a JSON array that can be piped directly into `jq`.
|
|
- `--num N` means "consume N records starting from the chosen offset". With the default `--offset end`, that means "wait for N new records from now".
|
|
- Use `--last N` when you want the most recent retained N records.
|
|
- Use `--offset start` to read from the beginning of retained history.
|
|
- Use `--output text` for the human-readable stream view or `--output jsonl` for one JSON object per line.
|
|
- Use `--value-only` to emit only decoded record values in JSON mode.
|
|
- Use `--timeout` when you want the script to stop automatically.
|
|
|
|
```bash
|
|
python3 scripts/ops/live_near_intents.py --last 10
|
|
python3 scripts/ops/live_near_intents.py --last 10 | jq '.[].value.payload.message.quote_id'
|
|
python3 scripts/ops/live_near_intents.py --num 10 --offset start
|
|
python3 scripts/ops/live_near_intents.py --num 10 --timeout 30
|
|
python3 scripts/ops/live_near_intents.py --value-only --last 5
|
|
python3 scripts/ops/live_near_intents.py --output text
|
|
```
|
|
|
|
### Near Intents control API
|
|
|
|
`near-intents-ingest` exposes a small in-process control API on port `8081` by
|
|
default. It is meant for ad hoc inspection and runtime filter changes without a
|
|
redeploy.
|
|
|
|
Port-forward the deployment:
|
|
|
|
```bash
|
|
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip port-forward deploy/near-intents-ingest 8081:8081
|
|
```
|
|
|
|
Inspect current state, including the active pair filter and ingest counters:
|
|
|
|
```bash
|
|
curl -s http://127.0.0.1:8081/state
|
|
```
|
|
|
|
Set or disable the runtime pair filter:
|
|
|
|
```bash
|
|
curl -s -X PUT http://127.0.0.1:8081/pair-filter \
|
|
-H 'content-type: application/json' \
|
|
-d '{"pair":"nep141:btc.omft.near->nep141:eth.omft.near"}'
|
|
|
|
curl -s -X PUT http://127.0.0.1:8081/pair-filter \
|
|
-H 'content-type: application/json' \
|
|
-d '{"pair":null}'
|
|
```
|
|
|
|
Reset the runtime filter back to the configured env/file/default state:
|
|
|
|
```bash
|
|
curl -s -X POST http://127.0.0.1:8081/pair-filter/reset
|
|
```
|
|
|
|
## Agent workflow
|
|
|
|
This repo now carries a small tracked workflow layer for Codex or other agents.
|
|
It is meant to create pressure toward real product progress without introducing
|
|
heavy orchestration.
|
|
|
|
The key files are:
|
|
|
|
- `AGENTS.md` — hard rules for agent behavior in this repo
|
|
- `THESIS.md` — stable product intent and approval boundaries
|
|
- `PROOF.md` — the active implementation proof
|
|
- `IMPLEMENTATION.md` — the current implementation turn
|
|
- `research/ACTIVE.md` — the active research charter
|
|
- `BACKLOG.md` — parked ideas, bugs, and future turn candidates
|
|
- `ARCHIVE.md` — index of archived turns and planning events
|
|
- `workflow/REVIEW_PROMPT.md` — adversarial review prompt for a separate review-only run
|
|
|
|
Two important rules shape the workflow:
|
|
|
|
- quote collection and analytics are first-class from day one
|
|
- backlog items do not automatically become active implementation without an explicit turn-opening step
|
|
|
|
### Install the tracked git hook
|
|
|
|
Install the tracked hook path once per clone:
|
|
|
|
```bash
|
|
bash scripts/workflow/install_hooks.sh
|
|
```
|
|
|
|
That sets `core.hooksPath` to `.githooks`.
|
|
|
|
The commit hook rejects non-merge commits unless the commit message body
|
|
contains:
|
|
|
|
- `Proof: ...`
|
|
- `Assumptions: ...`
|
|
- `Still fake: ...`
|
|
|
|
### Workflow scripts
|
|
|
|
`scripts/workflow/add_backlog.py`
|
|
- Append a new idea, bug, research item, or ops task to `BACKLOG.md`.
|
|
- Prints the created stable backlog ID.
|
|
|
|
```bash
|
|
python3 scripts/workflow/add_backlog.py --lane implementation --summary "Durable sink for normalized events"
|
|
python3 scripts/workflow/add_backlog.py --lane research --summary "Test whether quote freshness predicts worse downstream execution"
|
|
python3 scripts/workflow/add_backlog.py --lane bug --summary "Replay output drops pair metadata"
|
|
```
|
|
|
|
`scripts/workflow/open_turn.py`
|
|
- Opens a new implementation or research turn.
|
|
- Pulls selected backlog items into the active turn and removes them from `BACKLOG.md`.
|
|
- Refuses to overwrite an already-open turn unless `--force` is passed.
|
|
- Use `--commit` if you want the planning change committed automatically.
|
|
|
|
```bash
|
|
python3 scripts/workflow/open_turn.py \
|
|
--lane implementation \
|
|
--title "bounded replay for durable quote history" \
|
|
--summary "Replay recent history for the configured pair from the durable store." \
|
|
--pick I002 \
|
|
--pick B001
|
|
```
|
|
|
|
`scripts/workflow/close_turn.py`
|
|
- Archives the current implementation or research turn into `archive/`.
|
|
- Resets the live turn file back to `idle`.
|
|
- Updates `ARCHIVE.md`.
|
|
- Use `--commit` if you want the archive change committed automatically.
|
|
|
|
```bash
|
|
python3 scripts/workflow/close_turn.py \
|
|
--lane implementation \
|
|
--status passed \
|
|
--summary "Durable history landed in cluster storage and replay works for recent windows."
|
|
```
|
|
|
|
Possible close statuses are:
|
|
|
|
- `passed`
|
|
- `failed`
|
|
- `paused`
|
|
- `abandoned`
|
|
|
|
`scripts/workflow/review_diff.sh`
|
|
- Builds a review bundle consisting of a git diff plus the adversarial review prompt.
|
|
- Intended for a separate review-only agent run.
|
|
|
|
```bash
|
|
bash scripts/workflow/review_diff.sh HEAD~1
|
|
bash scripts/workflow/review_diff.sh main...HEAD
|
|
```
|
|
|
|
### Current workflow state
|
|
|
|
At the moment, the seeded active implementation proof is in:
|
|
|
|
- `PROOF.md`
|
|
- `IMPLEMENTATION.md`
|
|
|
|
That proof is focused on the first real quote -> reference-price -> decision ->
|
|
execute loop for the live configured pair, with PostgreSQL as the first durable
|
|
audit and analytics store behind the Kafka backbone.
|