# Trading System Architecture Notes for Next Session

## Objective
Build the first real version of the trading system as an event-driven, multi-service architecture.

Current implemented seed:
- NEAR Intents ingest in Node.js
- Kafka-compatible bus usage via `kafkajs`
- dummy reactor / executor / result consumer loop

Next session should continue from this architecture, not revert to a monolith, local-only script, or TUI.

---

## Core Architecture
All components are independent services.
They communicate only through a central Kafka-compatible bus (Redpanda first, Kafka-compatible by design).

### Service classes
- venue ingestors
- normalizers
- reactors / decision engines
- executors
- downstream consumers / monitors / archivers / replay tools

### Service communication rule
No direct service-to-service calls for core trading flow.
Use bus topics only.

---

## Venue-Oriented Structure
The system should be organized by venue.
Each venue can have different:
- ingest/feed mechanics
- normalization logic
- execution mechanics

### Per-venue responsibilities
- `ingest` = venue-native intake
- `normalize` = convert venue-native payload into canonical internal event
- `execute` = venue-specific action logic

Planned shape:
```text
src/
  apps/
  bus/
  core/
  venues/
    near-intents/
      ingest
      normalize
      execute
```

---

## Bus Choice
Use **Redpanda** first, but stay fully **Kafka-compatible**.

### Reason
Requirements:
- high throughput
- low latency
- retention
- replay
- multiple producers/consumers
- independent services
- future scale-out
- multi-language compatibility

### Constraint
Do not use broker-specific features that make migration to Kafka difficult.
Use standard Kafka clients and semantics.

---

## Data Model Principles
Kafka/Redpanda is the operational event backbone.

### Event model rules
- append-only
- immutable events
- versioned schemas
- raw and normalized events both preserved

### Every event should include
- `event_id`
- `event_type`
- `venue`
- `observed_at` / `ingested_at`
- `schema_version`
- `payload`
- optionally raw/original payload where appropriate

### Raw vs normalized
Keep both.
- raw topics = exact venue-native source truth
- normalized topics = canonical research/trading inputs

This is required for:
- replay
- debugging
- future backtesting
- future Spark/batch processing

---

## Current/Planned Topic Flow
Minimal 3-stage pipeline:

1. ingest publishes normalized demand
2. reactor publishes trade command
3. executor publishes trade result

### Topic classes
- `raw.*` = raw venue-native events
- `norm.*` = canonical normalized market events
- `cmd.*` = execution commands
- `exec.*` = execution outcomes
- later `signal.*` if needed for reactor outputs before command stage

### Current minimal topics
- `norm.swap_demand`
- `cmd.execute_trade`
- `exec.trade_result`

### NEAR Intents
NEAR Intents source currently feeds quote-demand style events from solver-bus websocket.
This is a venue ingest source, not the whole trading system.

---

## Execution Safety / Zero Downtime Requirements
This is critical.

### Constraint
Multiple executors must never duplicate the same trade/action during deploys, restarts, or rebalances.

### Must-have rules
1. Every execution command must carry a unique `command_id`
2. Commands must include deterministic idempotency information
3. Executors must be idempotent
4. Executors must belong to a consumer group per executor role
5. Commands should be partitioned by a stable execution key where ordering matters
6. Executor state must be persisted durably enough to detect duplicate command execution

### Kafka consumer groups are not sufficient alone
They help assign work, but they do not guarantee no duplicate processing under restart/rebalance conditions.
Idempotency is still required.

### Rolling updates / zero downtime
Executors must support:
- graceful shutdown
- stop taking new work before exit
- finish or safely recover in-flight work
- commit offsets only after safe execution state transition

### Persistence implication
Executor idempotency state is not optional metadata.
It is operational state that must survive pod restarts.

Current single-node k3s direction:
- executor state lives at `/var/lib/unrip/executor-state`
- Kubernetes mounts that path through persistent storage
- the Hetzner single-node overlay currently targets k3s `local-path` storage
- node loss without storage migration means duplicate-suppression history is lost

---

## Deployment Target
### First deployment phase
- single machine on Hetzner
- but still multiple independent services
- no architecture shortcuts that prevent future clustering

### Future target
- split across multiple machines
- cluster capable
- fault tolerant
- multi-node
- zero-downtime deploys

### Deployment rules from day 1
- every component is a separate container/service
- all config via env/config files
- communication over network/bus only
- persistent components use mounted volumes/PVCs
- no manual SSH-based operational workflow

---

## Infrastructure / Ops Direction
Target environment:
- Hetzner
- self-hosted CI/CD
- provisioning by code
- no GitHub dependency

### Desired stack direction
- Terraform for Hetzner provisioning
- Kubernetes-oriented target from the start
- self-hosted Git + CI/CD
- Kafka-compatible broker
- object storage later for long-term archived event history

### Single-node first, future cluster later
The first version may run on one machine, but deployment structure should already match a future distributed system.

### Current canonical operator path
After the repo split, the primary deployment path is shared across two repos:
- the separate platform repo owns Hetzner/OpenTofu, cloud-init, k3s bootstrap,
  Forgejo, the runner, and the shared registry
- this repo owns the app image, app manifests, and the app rollout workflow

#### Phase 0: platform bootstrap (platform repo)
1. A local operator workstation prepares platform bootstrap secrets in the
   platform repo.
2. The operator runs the platform repo bootstrap flow.
3. Terraform provisions the server, firewall, network, and cloud-init user-data.
4. cloud-init installs k3s automatically and prepares persistence directories
   plus bootstrap artifacts.
5. The workstation writes the public and in-cluster kubeconfigs.
6. The workstation injects the shared platform secrets and applies the shared
   platform manifests.
7. Forgejo and the runner come online in the cluster.

#### Phase 1: app repo handoff (this repo)
1. The operator creates this app repo's Kubernetes secrets such as
   `unrip-secrets` and `unrip-registry-creds`.
2. This repo is pushed or mirrored into Forgejo.
3. On push to `main`, the Forgejo runner applies `deploy/k8s/base`, builds the
   app image in-cluster with Kaniko, and updates the `unrip` deployments.
4. Terraform remains the infra mutation entrypoint, but app rollout is owned by
   this repo's workflow.

### Failure-recovery expectation
The overall bootstrap path must be rerunnable from the workstation.
Docs should keep treating recovery as:
- fix local secrets/inputs
- rerun the platform repo bootstrap script
- inspect the cluster with the generated kubeconfig
- destroy/recreate infra from the platform repo only when required

### Current repo-state caveats
The direction is clear, but the implementation still has caveats:
- this repo now assumes a pre-bootstrapped platform repo/cluster
- kubeconfig/auth handling is only as strong as the external Forgejo secret management
- base manifests still carry a placeholder bootstrap image until CI rolls the first real tag
- app secrets and registry pull credentials still require operator setup
- local Compose remains in the repo for development/testing, not as the canonical production path

### Minimal repo layout target
```text
deploy/
  hetzner/
    README.md
  k8s/
    base/
    overlays/
      hetzner-single-node/
infra/
  terraform/
    hetzner/
```

Guidelines:
- `infra/terraform/hetzner/` owns VM, firewall, networking, and cloud-init rendering
- `deploy/k8s/` owns Kubernetes-native manifests and overlays
- app runtime manifests should remain Kubernetes-native so they can later move from single-node k3s to a larger cluster with minimal rewrite
- secret material must not live in git in plaintext; bootstrap docs should describe workstation-driven injection or generated secret references

---

## Local Development / Testing Direction
Do not assume manual multi-terminal operation long term.

### Requirement
Need an orchestrated local/dev runtime.

### Local dev should preserve real boundaries
- separate services
- broker present
- env/config driven
- same event flow as production

### Current local/dev answer
Compose is still acceptable for:
- developer laptops
- fast local iteration
- debugging event flow
- validating container boundaries before Kubernetes rollout

But Compose should remain explicitly secondary to the repo-driven Hetzner + k3s path for production operations.

### Testing layers
1. unit tests for normalizers / schema logic / helpers
2. integration tests against Kafka-compatible broker
3. replay/simulation tests using retained event streams

---

## Spark Readiness
Do not add Spark now.
But keep the system Spark-compatible later by:
- preserving raw events
- preserving normalized events
- using immutable append-only event streams
- versioning schemas
- separating operational event log from future analytical processing

Spark later would be for:
- large-scale backtesting
- feature generation
- archive processing
- multi-venue analytics

---

## Immediate Next Engineering Tasks
Next session should focus on the following.

### 1. Clean current repo structure
Remove duplicate/legacy paths and keep one canonical structure only.

### 2. Keep/complete the 3-stage loop
- NEAR Intents ingest -> `norm.swap_demand`
- dummy reactor -> `cmd.execute_trade`
- dummy executor -> `exec.trade_result`
- downstream result consumer

### 3. Define canonical schemas
Define concrete event schemas for:
- normalized swap demand
- execute trade command
- trade result

### 4. Define executor idempotency model
Specify:
- `command_id`
- idempotency key rules
- execution state transition rules
- duplicate handling rules

### 5. Move toward production-shaped deployment
Design for:
- one service per container
- single-node deployment first
- future multi-node split without app rewrite

### 6. Harden provisioning/deployment path
Next infra work should continue improving:
- Hetzner provisioning by code
- workstation bootstrap rerunnability
- self-hosted CI/CD handoff
- registry-native image delivery
- overlay convergence for the Hetzner single-node target

Status update:
- minimal Terraform exists under `infra/terraform/hetzner`
- first boot is cloud-init driven and installs k3s automatically
- bootstrap now starts from a local operator workstation rather than manual host login
- Kubernetes assets exist under `deploy/k8s`
- executor persistence boundaries are explicit for single-node k3s
- self-hosted CI handoff is documented, but still requires follow-up hardening

---

## Non-Goals for Next Session
- no dashboards
- no UI/TUI
- no monolith convenience architecture
- no SQLite-first system of record
- no direct coupling between ingest, decision, and execution
- no temporary local-only shortcuts that block future cluster deployment

---

## Guiding Principle
Build the single-node first version as if it is already a distributed system:
- separate services
- durable event bus
- replayable events
- explicit contracts
- idempotent execution
- production-compatible deployment boundaries
- bootstrapable from scratch without manual SSH-based host setup