202 lines
5.4 KiB
Markdown
202 lines
5.4 KiB
Markdown
# near-intents-monitor
|
|
|
|
Production-shaped first slice of the trading system:
|
|
|
|
- **venue ingest**: NEAR Intents solver-bus quote flow
|
|
- **bus**: Redpanda first, Kafka-compatible by design
|
|
- **reactor**: dummy decision engine emitting commands
|
|
- **executor**: dummy execution worker with durable idempotency state
|
|
- **result consumer**: downstream observer of execution outcomes
|
|
|
|
## Canonical repo shape
|
|
|
|
```text
|
|
src/
|
|
apps/
|
|
near-intents-ingest.mjs
|
|
dummy-reactor.mjs
|
|
dummy-executor.mjs
|
|
dummy-consumer.mjs
|
|
bus/
|
|
kafka/
|
|
producer.mjs
|
|
consumer.mjs
|
|
core/
|
|
event-envelope.mjs
|
|
executor-state-store.mjs
|
|
log.mjs
|
|
pair-filter.mjs
|
|
schemas.mjs
|
|
lib/
|
|
config.mjs
|
|
env.mjs
|
|
venues/
|
|
near-intents/
|
|
ingest.mjs
|
|
normalize.mjs
|
|
ws.mjs
|
|
compose.yml
|
|
Dockerfile
|
|
docs/contracts.md
|
|
```
|
|
|
|
## Event flow
|
|
|
|
```text
|
|
NEAR Intents WebSocket
|
|
|
|
|
+--> raw.near_intents.quote
|
|
|
|
|
v
|
|
norm.swap_demand
|
|
|
|
|
v
|
|
cmd.execute_trade
|
|
|
|
|
v
|
|
exec.trade_result
|
|
```
|
|
|
|
Core rule: services do not call each other directly for trading flow; they communicate through bus topics only.
|
|
|
|
## Contracts
|
|
See `docs/contracts.md`.
|
|
|
|
Current topics:
|
|
- `raw.near_intents.quote`
|
|
- `norm.swap_demand`
|
|
- `cmd.execute_trade`
|
|
- `exec.trade_result`
|
|
|
|
## Canonical deployment path
|
|
|
|
The canonical production path is the repo-driven Hetzner + k3s bootstrap flow.
|
|
Compose still exists for local development and optional single-machine testing, but it is not the primary production story.
|
|
|
|
Current single-node cluster stack includes:
|
|
- `unrip` workloads in namespace `unrip`
|
|
- Redpanda
|
|
- Forgejo
|
|
- Forgejo runner
|
|
- private registry
|
|
- cert-manager
|
|
- Traefik via the k3s bundled ingress controller
|
|
- Grafana
|
|
- Loki
|
|
- Promtail
|
|
- Headlamp
|
|
|
|
### Bootstrap entrypoint
|
|
|
|
```bash
|
|
cp scripts/hetzner/bootstrap-secrets.env.example scripts/hetzner/bootstrap-secrets.env
|
|
source scripts/hetzner/bootstrap-secrets.env
|
|
bash scripts/hetzner/bootstrap.sh
|
|
```
|
|
|
|
The bootstrap script now:
|
|
1. provisions or updates Hetzner infra with Terraform
|
|
2. optionally manages DNS via Cloudflare or Porkbun
|
|
3. prefers Tailscale for admin/control-plane access when configured
|
|
4. fetches kubeconfig from the node into `.state/hetzner/kubeconfig.yaml`
|
|
5. renders `.state/hetzner/generated-overlay/` from repo manifests plus local secrets
|
|
6. applies platform and project resources to k3s
|
|
7. bootstraps Forgejo admin, runner, repo, and Actions configuration
|
|
8. seeds this repo into Forgejo
|
|
9. lets Forgejo Actions perform the default build/push/deploy path
|
|
10. stores the generated Headlamp login token in `pass` when `HEADLAMP_ADMIN_TOKEN_PASS` is configured
|
|
|
|
Detailed bootstrap and destroy documentation lives in:
|
|
- `docs/hetzner-k3s-bootstrap.md`
|
|
- `docs/hetzner-self-hosted-ci-runbook.md`
|
|
- `docs/k8s-observability.md`
|
|
- `deploy/hetzner/README.md`
|
|
- `deploy/k8s/README.md`
|
|
- `deploy/k8s/overlays/hetzner-single-node/README.md`
|
|
|
|
### Runtime surfaces
|
|
|
|
- Forgejo: `https://git.doran.133011.xyz/`
|
|
- Registry: `https://registry.doran.133011.xyz/`
|
|
- Grafana: `https://grafana.doran.133011.xyz/`
|
|
- Headlamp: `https://headlamp.doran.133011.xyz/`
|
|
|
|
### Operator notes
|
|
|
|
- Ingress is Traefik-based. The old ingress-nginx path is obsolete.
|
|
- Grafana is for historical log search.
|
|
- Headlamp is for browsing workloads, pods, events, and pod logs.
|
|
- Use `pass`-backed `*_PASS` variables for secrets whenever possible.
|
|
|
|
## Executor persistence in k3s
|
|
|
|
The executor is stateful by design because it persists idempotency/execution tracking.
|
|
|
|
Current persistence boundary:
|
|
- app env uses `EXECUTOR_STATE_DIR=/var/lib/unrip/executor-state`
|
|
- in Kubernetes, the executor deployment mounts storage at that path
|
|
- the Hetzner single-node overlay pins storage to the k3s `local-path` storage class
|
|
|
|
Operational meaning:
|
|
- executor state lives on node-backed storage in the single-node k3s environment
|
|
- if that PVC or underlying node storage is lost, duplicate-suppression history is lost too
|
|
- treat executor persistence as part of the minimal durable state of the cluster
|
|
|
|
## Local development with Compose
|
|
|
|
Compose remains available for local development and debugging.
|
|
|
|
```bash
|
|
npm install
|
|
cp .env.example .env
|
|
# edit .env
|
|
|
|
docker compose build
|
|
docker compose up -d
|
|
```
|
|
|
|
Useful commands:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs -f
|
|
docker compose logs -f near-intents-ingest dummy-reactor dummy-executor dummy-consumer
|
|
docker compose restart dummy-executor
|
|
docker compose down
|
|
docker compose down -v
|
|
```
|
|
|
|
### Individual services
|
|
```bash
|
|
npm run near-intents:ingest
|
|
npm run dummy-reactor
|
|
npm run dummy-executor
|
|
npm run dummy-consumer
|
|
```
|
|
|
|
Optional pair filter:
|
|
```bash
|
|
npm run near-intents:ingest -- --pair 'asset_a->asset_b'
|
|
```
|
|
|
|
## Idempotent executor behavior
|
|
- every command has a `command_id`
|
|
- commands carry `idempotency_key` and `execution_key`
|
|
- executor persists state under `EXECUTOR_STATE_DIR`
|
|
- completed commands are skipped after restart or replay
|
|
|
|
## Env
|
|
|
|
```env
|
|
NEAR_INTENTS_API_KEY=your_solver_jwt
|
|
NEAR_INTENTS_WS_URL=wss://solver-relay-v2.chaindefuser.com/ws
|
|
KAFKA_BROKERS=redpanda:9092
|
|
KAFKA_CLIENT_ID=unrip
|
|
KAFKA_TOPIC_RAW_NEAR_INTENTS_QUOTE=raw.near_intents.quote
|
|
KAFKA_TOPIC_NORM_SWAP_DEMAND=norm.swap_demand
|
|
KAFKA_TOPIC_CMD_EXECUTE_TRADE=cmd.execute_trade
|
|
KAFKA_TOPIC_EXEC_TRADE_RESULT=exec.trade_result
|
|
KAFKA_CONSUMER_GROUP_DUMMY=dummy-reactor-v1
|
|
KAFKA_CONSUMER_GROUP_EXECUTOR=dummy-executor-v1
|
|
EXECUTOR_STATE_DIR=/var/lib/unrip/executor-state
|
|
```
|