Commit graph

4 commits

Author SHA1 Message Date
philipp
82017dd301 Guard quote ingest against node OOM
Some checks failed
deploy / deploy (push) Failing after 47s
Proof: Live investigation showed doran-1 entered NodeNotReady with kubelet SystemOOM and TLS/control-plane timeouts while near-intents-ingest, history-writer, and operator-dashboard were the largest Node memory consumers. This commit adds websocket publish backpressure for the raw quote firehose and pod memory guardrails for the affected services.

Assumptions: Dropping quote frames while Kafka publishing is backpressured is safer than allowing unbounded in-flight publishes to take down the single-node cluster; retained Kafka/Postgres history remains best-effort under overload until the platform has enough capacity for full raw retention.

Still fake: This does not add durable queue spillover for skipped raw websocket frames, does not resize the node, and does not prove fee-complete trading PnL.
2026-05-13 18:25:54 +02:00
philipp
8641c60ab7 Suppress duplicate relay websocket close errors
All checks were successful
deploy / deploy (push) Successful in 33s
Proof: npm test; npm run operator-dashboard:build; node --test test/near-intents-ws.test.mjs test/solver-relay-ws.test.mjs; PYTHONPATH=. python3 test/bootstrap_script_static_test.py; PYTHONPATH=. python3 test/render_release_manifest_test.py; PYTHONPATH=. python3 test/repo_deployments_test.py; PYTHONPATH=. python3 test/ntfy_manifest_test.py; kubectl kustomize deploy/k8s/base.

Assumptions: nested error events emitted during WebSocket close are the same close attempt, not independent evidence that should be logged twice.

Still fake: Notification emissions are limited to credited deposits, completed withdrawals, and completed trades with durable inventory movement; generic alert notification policy remains disabled.
2026-04-16 14:36:18 +02:00
philipp
ea0a7cbb4c Fix bootstrap image preservation and ingest websocket close recursion
All checks were successful
deploy / deploy (push) Successful in 32s
Proof: npm test; npm run operator-dashboard:build; node --test test/near-intents-ws.test.mjs; bash -n scripts/deploy/bootstrap.sh; PYTHONPATH=. python3 test/bootstrap_script_static_test.py; PYTHONPATH=. python3 test/render_release_manifest_test.py; PYTHONPATH=. python3 test/repo_deployments_test.py; PYTHONPATH=. python3 test/ntfy_manifest_test.py; kubectl kustomize deploy/k8s/base.

Assumptions: bootstrap should preserve an existing deployed release image when refreshing app secrets; websocket close/error events may be reentrant in the Node runtime.

Still fake: Notification emissions are limited to credited deposits, completed withdrawals, and completed trades with durable inventory movement; generic alert notification policy remains disabled.
2026-04-16 14:28:54 +02:00
philipp
d5a7325e48 Reconnect quote ingest after websocket errors
All checks were successful
deploy / deploy (push) Successful in 32s
Proof: NEAR Intents quote ingest now schedules reconnect on websocket error even when the runtime does not emit close; regression test, dashboard build, and full npm test pass.

Assumptions: The observed live ingest outage after the 0.49 percent rollout was caused by the startup socket_error path leaving the websocket disconnected; reconnecting preserves the existing pair filter and topics without changing trading size or funds exposure.

Still fake: Venue-native terminal fill events, fee attribution, realized per-trade PnL, and full inventory-skew strategy controls remain incomplete.
2026-04-12 17:30:19 +02:00