337 lines
7.5 KiB
Markdown
337 lines
7.5 KiB
Markdown
# Google Drive Offload
|
|
|
|
Status: valid
|
|
|
|
This document covers Checkpoint 7: offloading closed raw collector files and
|
|
manifests to Google Drive with `rclone`.
|
|
|
|
This checkpoint does not prove production readiness or 24/7 reliability. A real
|
|
small upload must be run with a configured remote, and the later 24h soak test
|
|
must still pass.
|
|
|
|
## Scope
|
|
|
|
Included:
|
|
|
|
- `scripts/upload_archive_rclone.sh`
|
|
- `scripts/purge_uploaded_local_files.sh`
|
|
- `systemd/polymarket-orderbook-uploader.service`
|
|
- `systemd/polymarket-orderbook-uploader.timer`
|
|
- dry-run mode by default
|
|
- real upload only with `--execute`
|
|
- rclone verification with `rclone check`
|
|
- per-run upload manifests
|
|
- verified-upload index tracking
|
|
- periodic local purge of previously verified files
|
|
|
|
Excluded:
|
|
|
|
- dashboards
|
|
- databases
|
|
- strategies or backtests
|
|
- trading, signing, order placement, or wallet logic
|
|
- hardcoded private auth material
|
|
|
|
## Install rclone
|
|
|
|
On Ubuntu or Debian:
|
|
|
|
```sh
|
|
sudo apt-get update
|
|
sudo apt-get install -y rclone
|
|
```
|
|
|
|
Confirm:
|
|
|
|
```sh
|
|
rclone version
|
|
```
|
|
|
|
## Configure A Google Drive Remote
|
|
|
|
Configure the remote outside this repository. For a service-user setup:
|
|
|
|
```sh
|
|
sudo -u orderbooks rclone config
|
|
sudo -u orderbooks rclone lsd gdrive:
|
|
```
|
|
|
|
The example remote path is:
|
|
|
|
```text
|
|
gdrive:orderbooks/polymarket
|
|
```
|
|
|
|
Any valid `rclone` destination may be used. The uploader reads it from:
|
|
|
|
```text
|
|
ORDERBOOKS_RCLONE_DEST
|
|
```
|
|
|
|
For systemd, create:
|
|
|
|
```text
|
|
/etc/orderbooks/orderbook-uploader.env
|
|
```
|
|
|
|
Example:
|
|
|
|
```text
|
|
ORDERBOOKS_RCLONE_DEST=gdrive:orderbooks/polymarket
|
|
```
|
|
|
|
Do not commit the machine-local rclone config or any private auth material.
|
|
|
|
## What Gets Uploaded
|
|
|
|
By default the script targets:
|
|
|
|
| Source | Default path |
|
|
| --- | --- |
|
|
| raw collector files | `/var/lib/orderbooks/raw_orderbooks` |
|
|
| collector manifests | `/var/lib/orderbooks/manifests` |
|
|
|
|
It does not target normalized sample files by default.
|
|
|
|
Files modified within the last 10 minutes are skipped to avoid active collector
|
|
files:
|
|
|
|
```text
|
|
ORDERBOOKS_UPLOAD_MIN_AGE_SECONDS=600
|
|
```
|
|
|
|
The script preserves repository/data-directory relative paths on the remote. For
|
|
example:
|
|
|
|
```text
|
|
/var/lib/orderbooks/raw_orderbooks/polymarket/orderbooks/<run_id>/file.jsonl.gz
|
|
```
|
|
|
|
uploads to:
|
|
|
|
```text
|
|
<remote>/raw_orderbooks/polymarket/orderbooks/<run_id>/file.jsonl.gz
|
|
```
|
|
|
|
## Dry Run
|
|
|
|
Dry-run is the default. It plans files, stages a temporary copy, invokes
|
|
`rclone copy --dry-run`, and writes an upload manifest.
|
|
|
|
Example for a VPS:
|
|
|
|
```sh
|
|
/opt/orderbooks/scripts/upload_archive_rclone.sh \
|
|
--data-dir /var/lib/orderbooks \
|
|
--dest "$ORDERBOOKS_RCLONE_DEST"
|
|
```
|
|
|
|
Example against the repository sample data:
|
|
|
|
```sh
|
|
scripts/upload_archive_rclone.sh \
|
|
--data-dir data \
|
|
--dest gdrive:orderbooks/polymarket/checkpoint7-test \
|
|
--manifest-path data/manifests/upload_archive_real_test_dry_run_manifest.json \
|
|
--min-age-seconds 0 \
|
|
--rclone-bin /usr/bin/rclone
|
|
```
|
|
|
|
Dry-run does not prove remote write access.
|
|
|
|
## Execute Upload
|
|
|
|
Run a real upload only after the remote is configured and the dry-run plan looks
|
|
right:
|
|
|
|
```sh
|
|
/opt/orderbooks/scripts/upload_archive_rclone.sh \
|
|
--execute \
|
|
--data-dir /var/lib/orderbooks \
|
|
--dest "$ORDERBOOKS_RCLONE_DEST"
|
|
```
|
|
|
|
The script runs:
|
|
|
|
```text
|
|
rclone copy <staged files> <remote> --checksum
|
|
rclone check <staged files> <remote> --one-way --checksum
|
|
```
|
|
|
|
The upload gate is `PASS` only when the copy succeeds and verification succeeds.
|
|
|
|
## Retention And Cleanup
|
|
|
|
Local files are kept by default, even after upload verification.
|
|
|
|
Immediate same-run cleanup requires an explicit flag:
|
|
|
|
```sh
|
|
/opt/orderbooks/scripts/upload_archive_rclone.sh \
|
|
--execute \
|
|
--cleanup-after-verify \
|
|
--retention-days 7 \
|
|
--data-dir /var/lib/orderbooks \
|
|
--dest "$ORDERBOOKS_RCLONE_DEST"
|
|
```
|
|
|
|
Cleanup deletes only files that were selected for upload, uploaded, verified, and
|
|
older than the retention window. The default retention window is 7 days.
|
|
|
|
The uploader also maintains a durable verified-upload index at:
|
|
|
|
```text
|
|
/var/lib/orderbooks/manifests/upload_verified_index.json
|
|
```
|
|
|
|
That index records files that have already passed `rclone copy` and
|
|
`rclone check`. The periodic purge step uses that index to delete previously
|
|
verified local files after the retention window, even when the current upload
|
|
run is not the one that first verified them.
|
|
|
|
Run the purge manually with:
|
|
|
|
```sh
|
|
/opt/orderbooks/scripts/purge_uploaded_local_files.sh \
|
|
--execute \
|
|
--data-dir /var/lib/orderbooks \
|
|
--retention-days 7
|
|
```
|
|
|
|
The periodic systemd/Kubernetes runtime runs upload and purge together.
|
|
|
|
## Upload Manifest
|
|
|
|
Each run writes a manifest such as:
|
|
|
|
```text
|
|
/var/lib/orderbooks/manifests/upload_archive_YYYYMMDDTHHMMSSZ.json
|
|
```
|
|
|
|
The manifest records:
|
|
|
|
- planned files
|
|
- attempted files
|
|
- dry-run files
|
|
- uploaded files
|
|
- verified files
|
|
- skipped open or recent files
|
|
- retained local files
|
|
- deleted local files
|
|
- SHA-256 checksums
|
|
- command mode
|
|
- start/end time
|
|
- rclone copy/check exit codes
|
|
- gate status
|
|
- verified-upload index update summary
|
|
|
|
Each purge run writes a separate manifest such as:
|
|
|
|
```text
|
|
/var/lib/orderbooks/manifests/purge_uploaded_local_YYYYMMDDTHHMMSSZ.json
|
|
```
|
|
|
|
The purge manifest records:
|
|
|
|
- verified-index path and record count
|
|
- eligible files older than retention
|
|
- deleted local files
|
|
- skipped files such as checksum mismatches
|
|
- retention configuration
|
|
- gate and operation status
|
|
|
|
For this repository, the sample manifest path is:
|
|
|
|
```text
|
|
data/manifests/upload_archive_sample_manifest.json
|
|
```
|
|
|
|
The verified Checkpoint 7 real-test manifest is:
|
|
|
|
```text
|
|
data/manifests/upload_archive_real_test_manifest.json
|
|
```
|
|
|
|
## systemd Timer
|
|
|
|
Install the unit files:
|
|
|
|
```sh
|
|
sudo install -o root -g root -m 0644 /opt/orderbooks/systemd/polymarket-orderbook-uploader.service /etc/systemd/system/polymarket-orderbook-uploader.service
|
|
sudo install -o root -g root -m 0644 /opt/orderbooks/systemd/polymarket-orderbook-uploader.timer /etc/systemd/system/polymarket-orderbook-uploader.timer
|
|
sudo systemctl daemon-reload
|
|
```
|
|
|
|
Create the environment file:
|
|
|
|
```sh
|
|
sudo install -o root -g orderbooks -m 0640 /dev/null /etc/orderbooks/orderbook-uploader.env
|
|
sudo editor /etc/orderbooks/orderbook-uploader.env
|
|
```
|
|
|
|
At minimum, set:
|
|
|
|
```text
|
|
ORDERBOOKS_RCLONE_DEST=gdrive:orderbooks/polymarket
|
|
```
|
|
|
|
Enable the timer:
|
|
|
|
```sh
|
|
sudo systemctl enable --now polymarket-orderbook-uploader.timer
|
|
```
|
|
|
|
Run one upload immediately:
|
|
|
|
```sh
|
|
sudo systemctl start polymarket-orderbook-uploader.service
|
|
```
|
|
|
|
That service now runs upload verification first and then runs the verified-file
|
|
purge step in the same timer cycle.
|
|
|
|
## Logs
|
|
|
|
Use the systemd journal:
|
|
|
|
```sh
|
|
sudo systemctl status polymarket-orderbook-uploader.service
|
|
sudo journalctl -u polymarket-orderbook-uploader.service -f
|
|
sudo systemctl list-timers polymarket-orderbook-uploader.timer
|
|
```
|
|
|
|
## Current Checkpoint 7 Result
|
|
|
|
Initial local validation was blocked when `rclone` was unavailable. That blocked
|
|
manifest remains at:
|
|
|
|
```text
|
|
data/manifests/upload_archive_sample_manifest.json
|
|
```
|
|
|
|
After `rclone` was configured as `/usr/bin/rclone` with remote `gdrive:`, a dry
|
|
run and one tiny real upload were run against:
|
|
|
|
```text
|
|
gdrive:orderbooks/polymarket/checkpoint7-test
|
|
```
|
|
|
|
The real upload manifest records `rclone copy` exit code 0 and `rclone check`
|
|
exit code 0:
|
|
|
|
```text
|
|
data/manifests/upload_archive_real_test_manifest.json
|
|
```
|
|
|
|
Current gate:
|
|
|
|
```text
|
|
PASS
|
|
```
|
|
|
|
## What Remains Unproven
|
|
|
|
- Long-run upload reliability.
|
|
- Interaction between hourly uploads and a 24h collector soak test.
|
|
- Long-run purge behavior under repeated intermittent `rclone check` failures.
|
|
- Production readiness.
|