orderbooks/docs/VPS_DEPLOYMENT.md

VPS Deployment

Status: valid

This document covers the Checkpoint 6 systemd runtime package for the raw Polymarket order-book collector.

It does not claim production readiness or 24/7 reliability. That remains gated on the later 24h soak test.

Scope

Included:

• systemd service for the raw collector cycle
• Python virtualenv setup
• service user and directory permissions
• configurable data directory
• discovery refresh before each collector cycle
• journal-based logs
• safe restart model for finite collector runs

Excluded:

• Google Drive offload
• rclone
• uploader scripts, services, or timers
• normalization changes
• dashboards
• databases
• strategies or backtests
• trading, order placement, signing, or wallet logic

Uploader service and timer units are intentionally deferred to Checkpoint 7.

Runtime Model

The systemd service runs:

/opt/orderbooks/scripts/run_polymarket_collector_cycle.sh

Each cycle:

1. Refreshes BTC market discovery into the configured data directory.
2. Runs scripts/collect_polymarket_orderbooks.py once.
3. Writes run-rotated raw gzip JSONL files.
4. Writes a per-cycle collector manifest.
5. Exits after the configured finite duration.

The unit uses Restart=always, so systemd starts the next cycle after the prior cycle exits or fails.

The example config uses a 300 second collection cycle. This is deliberately short because current BTC up/down markets are short-lived and the collector refreshes discovery only before a cycle starts. Do not increase the cycle beyond the practical market horizon unless the collector later learns to refresh market selection during a run.

Paths

Default VPS paths:

Purpose	Path
Application checkout	/opt/orderbooks
Python virtualenv	/opt/orderbooks/.venv
Service config	/etc/orderbooks/polymarket_collector.vps.yaml
Optional env override file	/etc/orderbooks/polymarket-orderbook-collector.env
Data directory	/var/lib/orderbooks
Discovery artifacts	/var/lib/orderbooks/discovery
Raw order-book output base	/var/lib/orderbooks/raw_orderbooks
Per-cycle manifests	/var/lib/orderbooks/manifests

Adjust these paths if the repository is installed somewhere other than /opt/orderbooks.

Environment Variables

The service defines safe defaults and can load overrides from:

/etc/orderbooks/polymarket-orderbook-collector.env

Supported variables:

Variable	Default	Meaning
ORDERBOOKS_APP_DIR	/opt/orderbooks	Repository checkout path.
ORDERBOOKS_DATA_DIR	/var/lib/orderbooks	Base directory for data files.
ORDERBOOKS_PYTHON	/opt/orderbooks/.venv/bin/python	Python interpreter.
ORDERBOOKS_COLLECTOR_CONFIG	/etc/orderbooks/polymarket_collector.vps.yaml	Collector config path.
ORDERBOOKS_DISCOVERY_DIR	$ORDERBOOKS_DATA_DIR/discovery	Discovery artifact directory.
ORDERBOOKS_OUTPUT_DIR	$ORDERBOOKS_DATA_DIR/raw_orderbooks	Collector output base directory.
ORDERBOOKS_MANIFEST_DIR	$ORDERBOOKS_DATA_DIR/manifests	Per-cycle manifest directory.
ORDERBOOKS_DISCOVERY_LIMIT	100	Gamma event page limit per discovery page.
ORDERBOOKS_DISCOVERY_MAX_PAGES	3	Discovery page cap per cycle.
ORDERBOOKS_DISCOVERY_TIMEOUT	15	Discovery request timeout in seconds.

Example override file:

ORDERBOOKS_DATA_DIR=/var/lib/orderbooks
ORDERBOOKS_DISCOVERY_MAX_PAGES=3

No API keys are required for this checkpoint.

Install On Ubuntu Or Debian

Run package and account setup as root or with sudo:

sudo apt-get update
sudo apt-get install -y git python3 python3-venv
sudo useradd --system --home /var/lib/orderbooks --shell /usr/sbin/nologin orderbooks
sudo mkdir -p /opt /etc/orderbooks /var/lib/orderbooks/discovery /var/lib/orderbooks/raw_orderbooks /var/lib/orderbooks/manifests

Install or update the repository under /opt/orderbooks. One option is:

cd /opt
sudo git clone <repo-url> orderbooks

If the checkout already exists:

cd /opt/orderbooks
sudo git pull --ff-only

Prepare permissions:

sudo chown -R root:root /opt/orderbooks
sudo chmod -R a+rX /opt/orderbooks
sudo chmod +x /opt/orderbooks/scripts/run_polymarket_collector_cycle.sh
sudo chown -R orderbooks:orderbooks /var/lib/orderbooks

Create the virtualenv:

cd /opt/orderbooks
sudo python3 -m venv .venv
sudo .venv/bin/python -m pip install --upgrade pip
sudo chown -R root:root .venv
sudo chmod -R a+rX .venv

The current Checkpoint 6 scripts use only the Python standard library.

Install the VPS config and service unit:

sudo install -o root -g root -m 0644 /opt/orderbooks/config/polymarket_collector.vps.example.yaml /etc/orderbooks/polymarket_collector.vps.yaml
sudo install -o root -g root -m 0644 /opt/orderbooks/systemd/polymarket-orderbook-collector.service /etc/systemd/system/polymarket-orderbook-collector.service

Review /etc/orderbooks/polymarket_collector.vps.yaml before starting the service. The example writes under /var/lib/orderbooks.

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now polymarket-orderbook-collector.service

Logs And Status

Use the systemd journal:

sudo systemctl status polymarket-orderbook-collector.service
sudo journalctl -u polymarket-orderbook-collector.service -f

Recent logs without following:

sudo journalctl -u polymarket-orderbook-collector.service --since "1 hour ago"

Output Files

Raw gzip JSONL files are written under:

/var/lib/orderbooks/raw_orderbooks/polymarket/orderbooks/<run_id>/

Per-cycle manifests are written under:

/var/lib/orderbooks/manifests/polymarket_orderbook_collector_<cycle_id>.json

Discovery artifacts are refreshed under:

/var/lib/orderbooks/discovery/

Restart And Stop Behavior

The unit uses:

Restart=always
RestartSec=30s
TimeoutStopSec=90s
KillSignal=SIGTERM
KillMode=control-group

The collector handles SIGTERM by finishing or timing out the current request, closing the gzip output, and writing the manifest. Every cycle writes to a new run directory, so closed files are not reopened by the next cycle.

Stop the service with:

sudo systemctl stop polymarket-orderbook-collector.service

Start it again with:

sudo systemctl start polymarket-orderbook-collector.service

Local Validation Without Starting The Service

These checks do not require root:

python3 -m py_compile scripts/discover_polymarket_btc_markets.py scripts/collect_polymarket_orderbooks.py
bash -n scripts/run_polymarket_collector_cycle.sh
python3 - <<'PY'
from pathlib import Path
from scripts.collect_polymarket_orderbooks import load_flat_yaml
cfg = load_flat_yaml(Path('config/polymarket_collector.vps.example.yaml'))
required = {
    'discovery_path',
    'output_dir',
    'manifest_path',
    'market_limit',
    'interval_seconds',
    'duration_seconds',
}
missing = sorted(required - set(cfg))
assert not missing, missing
assert cfg['duration_seconds'] > 0
print('config parse ok')
PY

If systemd tools are available locally:

systemd-analyze verify systemd/polymarket-orderbook-collector.service

The local machine may not have /opt/orderbooks or the orderbooks service user. Treat missing VPS path or user messages as deployment-environment warnings, not collector syntax failures.

Safe Upgrade

Stop the service, update files, rerun validation, then start the service:

sudo systemctl stop polymarket-orderbook-collector.service
cd /opt/orderbooks
sudo git pull --ff-only
sudo .venv/bin/python -m py_compile scripts/discover_polymarket_btc_markets.py scripts/collect_polymarket_orderbooks.py
sudo systemctl daemon-reload
sudo systemctl start polymarket-orderbook-collector.service

Do not remove existing data files during an upgrade. If a bad artifact is found, preserve it and label it invalid or deprecated with a replacement path when one exists.

Current Limits

• This package runs the existing raw collector; it does not add a daemon inside Python.
• The systemd loop is a restart model around finite collector cycles.
• It does not upload files.
• It does not prove long-run reliability.
• Production readiness remains blocked until discovery, raw collection, offload, and a documented 24h soak test all pass.