unrip/README.md

# unrip project

This repository contains the unrip trading-system code and its project-specific deployment assets.

## Contents

- `src/` — application code
- `package.json` / `package-lock.json` — Node package manifest
- `Dockerfile` / `.dockerignore` — app container build
- `.env.example` — local app runtime example
- `compose.yml` — local development stack
- `deploy/k8s/base/` — project-specific Kubernetes manifests
- `deploy/redpanda/rpk-topics.txt` — project topic reference
- `docs/` — project-specific design and contract docs
- `docs/system-architecture.md` — high-level architecture, data flow, persistence map, and Mermaid diagrams

## Local development

```bash
npm install
cp .env.example .env
# edit .env

docker compose up -d --build
```

Useful commands:

```bash
docker compose ps
docker compose logs -f
docker compose logs -f \
  near-intents-ingest market-reference-ingest liquidity-manager \
  inventory-sync history-writer strategy-engine trade-executor
npm run near-intents:ingest
npm run market-reference:ingest
npm run liquidity:manager
npm run inventory:sync
npm run history:writer
npm run strategy:engine
npm run trade:executor
```

## App image

The app image is now built from this directory.

Examples:

```bash
docker build -t unrip:dev .
```

## Kubernetes manifests

Project manifests live under:

- `deploy/k8s/base/`

The shared cluster/platform resources live in the separate infra repository.

## Deployment

This repo is the app-side deployment repo. The shared Hetzner/k3s bootstrap,
Forgejo runner, registry, and other platform services live in the separate
platform repo.

See `docs/deployment.md` for the full operator path.
See `docs/operator-runbook.md` for the live turn control and arming sequence.

### One-time app bootstrap

Bootstrap the app namespace, secrets, and Forgejo repo settings from this repo:

```bash
bash scripts/deploy/bootstrap.sh
```

That bootstrap also refreshes the local `forgejo` remote URL for HTTPS pushes
when it has enough auth material to do so.

By default, the script uses the adjacent platform checkout at `../unrip3` for:
- `kubeconfig.yaml`
- `kubeconfig.incluster.yaml`
- registry credentials
- the `NEAR_INTENTS_API_KEY` fallback from `../unrip3/.env`

If you are not using that local split, provide the values yourself via env vars
such as `KUBECONFIG_PATH`, `CI_KUBECONFIG_PATH`, `REGISTRY_HOST`,
`REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `NEAR_INTENTS_API_KEY`, and either
`FORGEJO_TOKEN` or `FORGEJO_ADMIN_USERNAME` / `FORGEJO_ADMIN_PASSWORD`.

### Routine deploy

After bootstrap, deployment is just a push to Forgejo `main`:

```bash
git push forgejo main
```

`.forgejo/workflows/deploy.yml` then:
- applies `deploy/k8s/base`
- builds the image from this repo root inside the cluster with Kaniko
- pushes it to the shared registry
- rolls the `unrip` deployments
- uses a fresh temporary runner workspace on each run, so reruns do not require
  manual cleanup on the Forgejo runner

### Observe rollout

```bash
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip get deploy,pods,job
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/near-intents-ingest
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/market-reference-ingest
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/liquidity-manager
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/inventory-sync
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/history-writer
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/strategy-engine
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip rollout status deploy/trade-executor
```

### Operator dashboard

The operator dashboard runs as the `operator-dashboard` service inside the
`unrip` namespace and is exposed through the cluster ingress:

```text
https://doran.133011.xyz/
```

Local port-forward access is still available for development or incident
debugging.

Start local access:

```bash
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml \
  kubectl -n unrip port-forward --address 0.0.0.0 svc/operator-dashboard 8090:8090
```

Then open:

```text
http://<your-host-or-vpn-ip>:8090/
```

The dashboard uses HTTP basic auth in production.

- Username: `admin`
- Password: `pass unrip/dashboard`

You can print the current password locally with:

```bash
pass unrip/dashboard
```

### Operator dashboard frontend dev server

For faster UI iteration against the live cluster backend, use the local React
dev server with hot reload:

```bash
npm install
npm run operator-dashboard:dev
```

That command:
- reads the dashboard password from `pass unrip/dashboard`
- starts a local port-forward to the deployed `svc/operator-dashboard`
- serves the local React dashboard frontend with Vite live reload
- proxies `/api` and `/ws` to the deployed cluster backend with auth attached

Then open:

```text
http://<your-host-or-vpn-ip>:5173/
```

Useful overrides:

```bash
OPERATOR_DASHBOARD_DEV_PORT=5174 npm run operator-dashboard:dev
OPERATOR_DASHBOARD_DEV_HOST=127.0.0.1 npm run operator-dashboard:dev
OPERATOR_DASHBOARD_DEV_UPSTREAM_PORT=18091 npm run operator-dashboard:dev
OPERATOR_DASHBOARD_DEV_BASIC_AUTH_PASSWORD='...' npm run operator-dashboard:dev
PLATFORM_REPO_DIR=/path/to/unrip3 npm run operator-dashboard:dev
```

For the plain dashboard forward, you can also use the repo helper:

```bash
npm run operator-dashboard:forward
```

### Auxiliary ops scripts

These scripts default to the same adjacent platform checkout as the deployment
bootstrap: `../unrip3/.state/hetzner/kubeconfig.yaml`. Override with
`KUBECONFIG_PATH`, `KUBECONFIG`, or `PLATFORM_REPO_DIR` if needed.

`scripts/ops/deployment_status.py`
- Shows current deployment readiness, pod uptime, restart counts, and mounted storage usage.
- Outputs JSON by default so it can be piped directly into `jq`.
- By default it shows the live deployment pods only.
- Use `--include-completed` to include completed Job pods.
- Use `--include-rootfs` if you also want a probe of `/` for pods without PVC-backed mounts.
- Use `--output table` for the human-readable table view.

```bash
python3 scripts/ops/deployment_status.py
python3 scripts/ops/deployment_status.py | jq '.pods[] | {name, uptime}'
python3 scripts/ops/deployment_status.py --include-completed
python3 scripts/ops/deployment_status.py --output table
```

`scripts/ops/redpanda_storage.py`
- Shows how much data Redpanda is currently storing for the unrip topics.
- Outputs JSON by default so it can be piped directly into `jq`.
- Includes per-topic local bytes, total bytes, segment counts, and the overall Redpanda data-path usage.
- Use `--all-topics` to inspect every visible topic, or `--topic` multiple times for a subset.
- Use `--output table` for the human-readable table view.

```bash
python3 scripts/ops/redpanda_storage.py
python3 scripts/ops/redpanda_storage.py | jq '.totals'
python3 scripts/ops/redpanda_storage.py --all-topics
python3 scripts/ops/redpanda_storage.py --topic raw.near_intents.quote --topic norm.swap_demand
python3 scripts/ops/redpanda_storage.py --output table
```

`scripts/ops/live_near_intents.py`
- Reads the raw NEAR quote stream entering Redpanda.
- Outputs clean JSON by default. Bounded reads return a JSON array that can be piped directly into `jq`.
- `--num N` means "consume N records starting from the chosen offset". With the default `--offset end`, that means "wait for N new records from now".
- Use `--last N` when you want the most recent retained N records.
- Use `--offset start` to read from the beginning of retained history.
- Use `--output text` for the human-readable stream view or `--output jsonl` for one JSON object per line.
- Use `--value-only` to emit only decoded record values in JSON mode.
- Use `--timeout` when you want the script to stop automatically.

```bash
python3 scripts/ops/live_near_intents.py --last 10
python3 scripts/ops/live_near_intents.py --last 10 | jq '.[].value.payload.message.quote_id'
python3 scripts/ops/live_near_intents.py --num 10 --offset start
python3 scripts/ops/live_near_intents.py --num 10 --timeout 30
python3 scripts/ops/live_near_intents.py --value-only --last 5
python3 scripts/ops/live_near_intents.py --output text
```

### Near Intents control API

`near-intents-ingest` exposes a small in-process control API on port `8081` by
default. It is meant for ad hoc inspection and runtime filter changes without a
redeploy.

Port-forward the deployment:

```bash
KUBECONFIG=../unrip3/.state/hetzner/kubeconfig.yaml kubectl -n unrip port-forward deploy/near-intents-ingest 8081:8081
```

Inspect current state, including the active pair filter and ingest counters:

```bash
curl -s http://127.0.0.1:8081/state
```

Set or disable the runtime pair filter:

```bash
curl -s -X PUT http://127.0.0.1:8081/pair-filter \
  -H 'content-type: application/json' \
  -d '{"pair":"nep141:btc.omft.near->nep141:eth.omft.near"}'

curl -s -X PUT http://127.0.0.1:8081/pair-filter \
  -H 'content-type: application/json' \
  -d '{"pair":null}'
```

Reset the runtime filter back to the configured env/file/default state:

```bash
curl -s -X POST http://127.0.0.1:8081/pair-filter/reset
```

## Agent workflow

This repo now carries a small tracked workflow layer for Codex or other agents.
It is meant to create pressure toward real product progress without introducing
heavy orchestration.

The key files are:

- `AGENTS.md` — hard rules for agent behavior in this repo
- `THESIS.md` — stable product intent and approval boundaries
- `PROOF.md` — the active implementation proof
- `IMPLEMENTATION.md` — the current implementation turn
- `research/ACTIVE.md` — the active research charter
- `BACKLOG.md` — parked ideas, bugs, and future turn candidates
- `ARCHIVE.md` — index of archived turns and planning events
- `workflow/REVIEW_PROMPT.md` — adversarial review prompt for a separate review-only run

Two important rules shape the workflow:

- quote collection and analytics are first-class from day one
- backlog items do not automatically become active implementation without an explicit turn-opening step

### Install the tracked git hook

Install the tracked hook path once per clone:

```bash
bash scripts/workflow/install_hooks.sh
```

That sets `core.hooksPath` to `.githooks`.

The commit hook rejects non-merge commits unless the commit message body
contains:

- `Proof: ...`
- `Assumptions: ...`
- `Still fake: ...`

### Workflow scripts

`scripts/workflow/add_backlog.py`
- Append a new idea, bug, research item, or ops task to `BACKLOG.md`.
- Prints the created stable backlog ID.

```bash
python3 scripts/workflow/add_backlog.py --lane implementation --summary "Durable sink for normalized events"
python3 scripts/workflow/add_backlog.py --lane research --summary "Test whether quote freshness predicts worse downstream execution"
python3 scripts/workflow/add_backlog.py --lane bug --summary "Replay output drops pair metadata"
```

`scripts/workflow/open_turn.py`
- Opens a new implementation or research turn.
- Pulls selected backlog items into the active turn and removes them from `BACKLOG.md`.
- Refuses to overwrite an already-open turn unless `--force` is passed.
- Use `--commit` if you want the planning change committed automatically.

```bash
python3 scripts/workflow/open_turn.py \
  --lane implementation \
  --title "bounded replay for durable quote history" \
  --summary "Replay recent history for the configured pair from the durable store." \
  --pick I002 \
  --pick B001
```

`scripts/workflow/close_turn.py`
- Archives the current implementation or research turn into `archive/`.
- Resets the live turn file back to `idle`.
- Updates `ARCHIVE.md`.
- Use `--commit` if you want the archive change committed automatically.

```bash
python3 scripts/workflow/close_turn.py \
  --lane implementation \
  --status passed \
  --summary "Durable history landed in cluster storage and replay works for recent windows."
```

Possible close statuses are:

- `passed`
- `failed`
- `paused`
- `abandoned`

`scripts/workflow/review_diff.sh`
- Builds a review bundle consisting of a git diff plus the adversarial review prompt.
- Intended for a separate review-only agent run.

```bash
bash scripts/workflow/review_diff.sh HEAD~1
bash scripts/workflow/review_diff.sh main...HEAD
```

### Current workflow state

At the moment, the seeded active implementation proof is in:

- `PROOF.md`
- `IMPLEMENTATION.md`

That proof is focused on the first real quote -> reference-price -> decision ->
execute loop for the live configured pair, with PostgreSQL as the first durable
audit and analytics store behind the Kafka backbone.