binance-trading-bot Documentation

File: README.md Copy Chapter

[!WARNING]
I cannot guarantee whether you can make money or not. Use it at your own risk. I have no responsibility for any loss or hardship incurred directly or indirectly by using this code. Read the disclaimer before using it.

Under active development and not recommended for real funds.

How it works Copy Chapter

You run the app on your own server, connect it to your Binance account with an API key, and it watches the market and places buy and sell orders on your behalf. Once running, it repeats a short loop for every coin you told it to watch: read the market, run your strategy's rules, decide buy / sell / wait, and act.

• Strategy-pluggable. A strategy is the rulebook the bot trades by. Three ship today: trailing-trade, momentum, and rebalance, and you pick one per profile. Each lives in its own package behind a shared contract, so adding a fourth is mostly a new package plus a registry entry, not a rewrite of the trading loop.
• Account-scoped profiles. One operator login owns one or more Binance accounts. Each account is one API key pair, one environment (testnet or live), and one wallet. Each account runs N independent profiles, each with its own coins, budget, and strategy, all sharing that account's wallet.
• Reliability-first. Crash-only worker, idempotent jobs, and a version-aware per-symbol state commit mean a restart resumes cleanly and never double-places an order.

Persistence is Postgres + TimescaleDB; cache and queues are Redis + BullMQ. Schema lives in packages/db.

Act on a symbol Copy Section

Every monitored symbol gets a workspace: live market data, the strategy's current signal, open orders, and manual overrides in one place.

Tune a strategy, then backtest it Copy Section

Configure a strategy per profile, then replay it against historical candles before risking anything. Every run is kept, so you can compare windows and symbols side by side.

More screens are covered in the user guide.

Configuration Copy Chapter

Per-profile configuration (grid, indicators, notifier providers, Technicals gates) lives in the database and is editable through the SPA after first run. See configure a profile.

Process-level environment variables (DATABASE_URL, REDIS_URL, WEB_ORIGIN, AUTH_SECRET, etc.) are listed in .env.example and described in the environment variable reference.

Quick start (development) Copy Chapter

Prerequisites: Bun 1.3+ and Docker (with the compose plugin) for the local Postgres + Redis stack.

Run all commands from the repo root.

# 1. Install deps, write .env from .env.example, bring up Postgres + Redis
#    with host ports exposed (via the local compose override), and run
#    migrations.
bun install
bun run setup

# 2. Start api + web + worker + technicals via turbo
bun run dev

Open the SPA at http://localhost:5173.

To run the whole stack in containers instead, use bun run docker:dev. One app service (ROLE=all) serves the SPA + /api on http://localhost:53000.

Quick start (deploy to a VM) Copy Chapter

The 9-step operator runbook lives at deploy/README.md. It covers .env, AUTH_SECRET, image pull, docker compose up, migrations, onboarding, IP-allowlisting the Binance key, smoke tests, and three TLS options.

Images are published to Docker Hub as chrisleekr/binance-trading-bot:vX.Y.Z. Pin a version with IMAGE_TAG rather than tracking :latest.

Documentation Copy Chapter

The full guide is at https://chrisleekr.github.io/binance-trading-bot, split by audience:

Build the site locally:

bun run docs:install
bun run docs:serve   # http://127.0.0.1:8000

Contributing Copy Chapter

Issues and pull requests are welcome at github.com/chrisleekr/binance-trading-bot.

1. Read CONTRIBUTING.md for the workflow.
2. Read AGENTS.md for the engineering charter and the invariants (CLAUDE.md is a symlink alias for Claude Code).
3. bun run lint and bun run typecheck must be clean before opening a pull request.

Security issues: please follow SECURITY.md rather than opening a public issue.

License Copy Chapter

Apache-2.0. Third-party attributions are in NOTICE.

Disclaimer Copy Chapter

I give no warranty and accept no responsibility or liability for the accuracy or the completeness of the information and materials contained in this project. Under no circumstances will I be held responsible or liable in any way for any claims, damages, losses, expenses, costs or liabilities whatsoever (including, without limitation, any direct or indirect damages for loss of profits, business interruption or loss of information) resulting from or arising directly or indirectly from your use of or inability to use this code or any code linked to it, or from your reliance on the information and material on this code, even if I have been advised of the possibility of such damages in advance.

So use it at your own risk!

File: deploy/README.md Copy Chapter

Operator deployment runbook

This document is the canonical first-run guide for an operator deploying binance-trading-bot to a private VM. The goal is deploy on a clean VM in under 30 minutes following the steps below.

The compose stack is intentionally edge-agnostic: it does not bundle a reverse proxy. Section TLS at the edge covers three ways to put TLS in front of the stack; pick one.

Prerequisites Copy Chapter

• A Linux VM with docker (≥ 24) and docker compose (≥ 2.20).
• 2 vCPU, 4 GB RAM, 20 GB disk is enough for one master account / a handful of profiles.
• Outbound HTTPS to api.binance.com and wss://stream.binance.com.
• A Binance Spot API key + secret. The key MUST have the trading permission and IP-allowlisting MUST be enabled (see Binance API key — IP allowlist).
• 32 random bytes for AUTH_SECRET (openssl rand -hex 32).

9-step runbook Copy Chapter

1. Clone the repository Copy Section

git clone https://github.com/chrisleekr/binance-trading-bot.git
cd binance-trading-bot

2. Copy `.env.example` Copy Section

cp .env.example .env

Edit .env at the repo root. For compose, set DATABASE_URL / REDIS_URL to the in-network DNS form (postgres://postgres:postgres@postgres:5432/binance_trading_bot, redis://redis:6379) or delete those lines to fall through to the compose file's built-in defaults; change WEB_ORIGIN to the URL the SPA will be reachable at.

3. Generate `AUTH_SECRET` Copy Section

For development, write the value into .env:

# Overwrite the AUTH_SECRET= line in-place
sed -i.bak "s|^AUTH_SECRET=.*|AUTH_SECRET=$(openssl rand -hex 32)|" .env \
  && rm .env.bak

For production, three independent secrets are needed: postgres password, redis password, and the api's AUTH_SECRET. The compose stack handles them three different ways:

Generate the three secret files (the postgres and redis containers consume them as mounts; the api needs the AUTH_SECRET line patched into .env):

mkdir -p deploy/secrets
openssl rand -hex 32 > deploy/secrets/session_secret
openssl rand -hex 32 > deploy/secrets/postgres_password
openssl rand -hex 32 > deploy/secrets/redis_password
chmod 600 deploy/secrets/*

# AUTH_SECRET and REDIS_URL are read from env by the api — use sed (not
# `>>`) so re-running this block stays idempotent and does not duplicate
# the keys in .env.
REDIS_PW=$(cat deploy/secrets/redis_password)
sed -i.bak \
  -e "s|^AUTH_SECRET=.*|AUTH_SECRET=$(cat deploy/secrets/session_secret)|" \
  -e "s|^REDIS_URL=.*|REDIS_URL=redis://:${REDIS_PW}@redis:6379|" \
  .env \
  && rm .env.bak
chmod 600 .env

The prod redis service enforces --requirepass; without the auth-aware REDIS_URL the api/worker fail to connect.

4. Pull (or build) the images Copy Section

A single image is published to Docker Hub by release-please (.github/workflows/release-please.yml) at chrisleekr/binance-trading-bot with an immutable version tag (vX.Y.Z); latest moves with each release on main. Pin a version in .env (IMAGE_TAG).

The Docker Hub repo is public, so unauthenticated pulls work. If you hit Docker Hub's anonymous rate limit, log in with any Docker Hub account first:

docker login

Then pull:

docker compose -f deploy/compose/docker-compose.yml \
               -f deploy/compose/docker-compose.prod.yml \
               --env-file .env pull

For air-gapped first-run or pre-release branches, build from source:

docker compose -f deploy/compose/docker-compose.yml build

5. Start the stack Copy Section

docker compose -f deploy/compose/docker-compose.yml \
               -f deploy/compose/docker-compose.prod.yml \
               --env-file .env up -d

The depends_on: { condition: service_healthy } chain blocks the app service until Postgres + Redis are accepting connections, so a fresh boot completes cleanly without retry loops.

6. Database migrations (automatic on boot) Copy Section

Migrations run automatically on boot: the container entrypoint (apps/server/docker-entrypoint.sh) runs the migration runner before starting the app, unless SKIP_MIGRATIONS=1. The runner is idempotent — every migration file is keyed by checksum in _app_migrations, so re-running is safe. A migration that fails exits the container non-zero and readiness never answers, so the orchestrator backs off rather than serving a stale schema. In the scale topology only the api service migrates; worker/study set SKIP_MIGRATIONS=1.

To run migrations manually (offline path), invoke the same binary:

docker compose exec app bun /app/dist/migrate.js

7. Open the URL — first-run onboarding Copy Section

Browse to WEB_ORIGIN. The first-run UI prompts you to create the master account (Better Auth, argon2id, no email). Once an account exists, subsequent visits to /onboarding redirect to the login page; password recovery requires the operator-side bun run reset-password CLI documented below.

8. IP-allowlist your Binance API key Copy Section

In the Binance API management UI, edit the key created for this bot and:

• Enable Restrict access to trusted IPs only.
• Add the public egress IP of your VM (use curl -s ifconfig.me from the VM to find it).
• Confirm the Enable Spot & Margin Trading permission is checked.

Without IP allow-listing, a leaked key dumps the entire account; this step is the threat model's primary mitigation since keys are stored plaintext in Postgres.

9. Verify healthchecks Copy Section

The api exposes /healthz, /readyz, and /metrics on a separate admin listener bound to 127.0.0.1:$ADMIN_PORT (default 9100) — the public network cannot reach them. Under ROLE=all (and ROLE=api) the app serves them. Hit them from the VM itself:

docker compose ps                              # all services should show (healthy)
curl -fsS http://127.0.0.1:9100/healthz        # api liveness
curl -fsS http://127.0.0.1:9100/readyz         # api readiness (DB + Redis ping)

If /readyz returns 503, follow the troubleshooting section below.

TLS at the edge Copy Chapter

The default stack serves plain HTTP on the published APP_HTTP_PORT (prod default 80 → container 3000); the app service serves both the SPA and /api same-origin, so there is no separate web edge. Production deployments MUST front the app service with TLS. Three reference configurations:

Option A — Cloudflare Tunnel Copy Section

Lowest-friction; no public ingress needed on the VM.

# On the VM, after `docker compose up -d`:
docker run -d --network=binance-trading-bot_internal \
  --name cloudflared --restart unless-stopped \
  cloudflare/cloudflared:latest tunnel \
  --no-autoupdate run --token <YOUR_TUNNEL_TOKEN>

In the Cloudflare dashboard, set the public hostname to forward to http://app:3000 (the in-network DNS name of the app service).

Option B — nginx / Traefik in front of compose Copy Section

If the operator already runs an nginx or Traefik on the host:

nginx fragment:

server {
  listen 443 ssl http2;
  server_name binance-trading-bot.example.com;
  ssl_certificate     /etc/letsencrypt/live/binance-trading-bot.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/binance-trading-bot.example.com/privkey.pem;

  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

  location / {
    # The `app` service, published on the host at APP_HTTP_PORT (prod default 80).
    proxy_pass         http://127.0.0.1:80;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto https;
    proxy_set_header   Upgrade           $http_upgrade;
    proxy_set_header   Connection        $http_connection;
    proxy_read_timeout 300s;
  }
}

Traefik fragment (in traefik.yml / dynamic config):

http:
  routers:
    binance-trading-bot:
      rule: 'Host(`binance-trading-bot.example.com`)'
      entryPoints: [websecure]
      tls: { certResolver: letsencrypt }
      service: binance-trading-bot
  services:
    binance-trading-bot:
      loadBalancer:
        servers:
          # The `app` service, published on the host at APP_HTTP_PORT (prod default 80).
          - url: 'http://127.0.0.1:80'

Either approach: point the proxy's upstream at the app service — in network http://app:3000, or on the host http://127.0.0.1:${APP_HTTP_PORT}. The app serves both the SPA and /api same-origin on container port 3000, so a single upstream keeps the Content-Security-Policy aligned.

Option C — Hosted reverse proxy (Vercel, Fly) Copy Section

For VMs without public ingress, run the SPA on a hosted edge that tunnels back to the VM.

• Vercel: deploy the apps/web build artifact to Vercel; configure rewrites so /api/* proxies to the VM via a Cloudflare Tunnel or a Tailscale endpoint. Set WEB_ORIGIN=https://<your-vercel>.vercel.app in .env.
• Fly.io: flyctl proxy 80:80 -a <fly-app> against a Fly app with a [[services]] block forwarding to the VM via Tailscale or wireguard. Same WEB_ORIGIN discipline.

The compose stack is unchanged in all three cases.

Restic off-host backup recipe Copy Chapter

The compose stack ships a nightly pg_dump loop (backup service, #66/13.06) into ./backups with 14-day local retention. Off-host shipping is the operator's choice; restic is the recommended tool.

Install Copy Section

# Ubuntu / Debian:
sudo apt install -y restic

# Or, latest binary:
curl -L https://github.com/restic/restic/releases/latest/download/restic_linux_amd64.bz2 \
  | bunzip2 -c | sudo tee /usr/local/bin/restic >/dev/null \
  && sudo chmod +x /usr/local/bin/restic

Initial setup Copy Section

Choose one storage backend; pick whatever the operator already has, then export the matching block (do not run all three back-to-back — RESTIC_REPOSITORY would be overwritten).

# Option 1 — AWS S3 / MinIO / any S3-compatible:
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export RESTIC_REPOSITORY=s3:s3.amazonaws.com/binance-trading-bot-backups

# Option 2 — Backblaze B2:
export B2_ACCOUNT_ID=...
export B2_ACCOUNT_KEY=...
export RESTIC_REPOSITORY=b2:binance-trading-bot-backups

# Option 3 — Storj (S3-compatible):
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=eu1
export RESTIC_REPOSITORY=s3:gateway.storjshare.io/binance-trading-bot-backups

Then set the password file and initialise the repo:

export RESTIC_PASSWORD_FILE=/root/.restic-password
echo 'change-me-32-bytes-of-entropy' > /root/.restic-password
chmod 600 /root/.restic-password

restic init

Daily cron Copy Section

# /etc/cron.daily/restic-binance-trading-bot
#!/bin/sh
set -eu
. /etc/restic.env       # exports the AWS_/B2_/RESTIC_* vars above
cd /path/to/binance-trading-bot
restic backup ./backups --tag binance-trading-bot --tag $(hostname)
restic forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --prune

Restore Copy Section

restic snapshots
restic restore <snapshot-id> --target /tmp/restore

# List the restored files — backup filenames are timestamped, so pick
# the dump you want (e.g. binance-trading-bot-YYYYMMDDTHHMMSSZ.dump).
ls /tmp/restore/backups/

# `./backups` (relative to the repo root) is already bind-mounted into
# the postgres container at `/backups`, so drop the restored dump in
# that host dir and pg_restore from there — no extra `docker cp` needed.
# Run from the repo root, or anchor explicitly with $(git rev-parse --show-toplevel).
REPO_ROOT=$(git rev-parse --show-toplevel)
cp "/tmp/restore/backups/<dump>.dump" "$REPO_ROOT/backups/restore.dump"
docker compose exec -T postgres pg_restore \
  -U postgres -d binance_trading_bot /backups/restore.dump

./backups contains plaintext Binance API keys (per the threat model; see security note). The restic password must be kept off-host (a password manager, not the VM disk) so a host compromise does not leak the off-host backups.

Sensitive data Copy Chapter

./backups/*.dump and postgres_data/ both contain plaintext Binance API keys (the design accepts this in exchange for IP-allowlisting at the Binance console). The operator MUST:

• chmod 700 ./backups deploy/secrets after the first run.
• Treat the off-host restic repo as a sensitive store (separate password file, not in this repo).
• Rotate Binance API keys if either the ./backups directory or the Postgres volume is exposed.

Troubleshooting Copy Chapter

Operator commands Copy Chapter

# Tail the app (api + live worker + study in one process)
docker compose logs -f app

# Run a one-shot backup
docker compose run --rm backup

# Reset the master password (offline)
docker compose run --rm app bun run reset-password

# Re-run DB migrations manually (they also run on boot via the entrypoint)
docker compose exec app bun /app/dist/migrate.js

# Stop everything (volumes preserved)
docker compose down

# Stop and wipe everything (DESTRUCTIVE)
docker compose down -v

File: docs/architecture/account-isolation.md Copy Chapter

Account isolation

Base-asset exclusivity (one base asset per Binance account) Copy Chapter

A base asset is managed by at most one profile per account. An account is now a first-class row (accounts): one Binance API key pair, one binance_mode (the environment that key pair talks to), and one user-data stream. The operator owns N accounts under a single login; every profile references its parent via profiles.account_id, and all profiles under one account share that account's single key pair, wallet, and stream. The base asset is the unit of exclusivity because it IS the shared wallet line: two profiles drawing on one BTC balance — whether via BTCUSDT, BTCFDUSD, or any other BTC* pair — cannot size a sell or arm a protective stop independently. One profile's resting stop locks tokens the other counts as free, leading to under-sized stops, failed sells, and stops placed over a position the other profile opened. Keying on the base asset subsumes the per-symbol check: blocking the base also blocks every cross-quote pair over it.

Each bound row carries a denormalized base_asset column (profile_symbols), backfilled by stripping the profile's quote_asset suffix. The denormalization lets the guard stay a total pure-SQL backstop: the repo cannot read Binance exchangeInfo, so every bind seam resolves the base from exchangeInfo and passes it into upsert.

Enforcement is at the single bind seam — profileSymbols.upsert (packages/db/src/repo/profile-symbols.ts) — which runs three exclusivity checks, base-owned first (it wins precedence): findOwningSiblingByBase throws SymbolOwnershipConflictError when another profile on the same account already manages the base asset, then findSiblingQuotingBase throws SiblingQuoteConflictError for the quote-collision half below, then a self-check rejects when the binding profile's own quote_asset (uppercased) equals the candidate base — settling in the very asset it trades as a base — also throwing SymbolOwnershipConflictError. Every add path funnels through upsert: discovery rotation, manual add, orphan-order adoption, and config reset. The API maps both errors to 409 CONFLICT. Discovery additionally pre-filters sibling-owned candidates (discovery.cron.ts) so the rotation loop stays exception-free. This is an app-level guard, not a DB unique index: the worker is single-replica and discovery serialises per profile, so the only theoretical race is an operator-driven concurrent API write.

Quote-collision exclusivity (base equals a sibling's quote asset) Copy Section

The shared wallet also couples in a second direction: a candidate whose base asset equals a sibling profile's quote asset. If a USDT profile buys BTCUSDT (base BTC) while a sibling profile settles in BTC (e.g. trades *BTC pairs), both draw on the one BTC balance line, so the sibling's sells and stops fight the same interference base-asset exclusivity guards against. Discovery auto-admission refuses such a candidate with the sibling-quotes-base disposition (its counterpart, base-owned-by-sibling, shows sibling-owns-base), surfaced in the Live universe so the operator sees why a matching coin was not added. Sibling quote assets are read account-scoped (profiles.listForAccount) once per profile per cron cycle. This half is also backstopped in upsert via findSiblingQuotingBase (comparing upper(quote_asset) against the candidate base, since a stored quote may be lower-case), throwing SiblingQuoteConflictError → 409 CONFLICT. So every bind seam — manual add, orphan adoption, config reset, and the redundant discovery re-add — is blocked, not just the discovery pre-filter. The discovery pre-filter remains as the exception-free fast path that shows the disposition; the DB backstop is what covers the manual and adoption routes the pre-filter never sees. The orphan-adoption route additionally pre-checks both halves before its order insert, so a conflict 409s without leaving a tracked order with no binding.

Editing a profile's own quote_asset is the same collision reached through a different door: pointing profile B's settlement at an asset a sibling already trades as a base (or that B itself trades) recreates the shared-wallet conflict after both are bound, and the symbol-bind guard never runs on a quote edit. So profiles.update runs the guard too, before it persists: when the new quote differs from the stored one it rejects a cross-profile clash (findOwningSiblingByBase) and a self-clash (profileManagesBase — B trading Q as a base while settling in Q), both throwing SymbolOwnershipConflictError → 409 CONFLICT. Only a genuine change is guarded, so re-setting the same quote never spuriously fails.

Two profiles under different accounts may hold the same base asset — a different account is a different key pair and wallet (and, since binance_mode lives on the account, possibly a different exchange environment), so there is no shared wallet line. The only residual is one profile binding two symbols over the SAME base (e.g. BTCUSDT and BTCFDUSD under one profile): the guard excludes self, so that intra-profile overlap is allowed and both rows track the one BTC balance — see reconcile-held-quantity.ts.

Destructive deletes (profile and account) Copy Chapter

Deleting a profile is a soft disposal that archives the profile and cancels all resting orders, rather than a row delete.

DELETE /api/accounts/:accountId/profiles/:profileId now takes a disposition and answers 202:

• no disposition + live exposure → 409 CONFLICT with { openOrderCount, openPositionCount }, so the UI can name what is open. (No exposure ⇒ no choice to make; it defaults to cancel-orders.)
• ?disposition=cancel-orders → the worker cancels every resting order on Binance, then deletes. The coins return to the wallet as plain holdings.
• ?disposition=handoff&toProfileId= → the orders are still cancelled (a clientOrderId encodes the SOURCE profile, so the target's strategy could never recognise them — it would see a foreign resting SELL and refuse to arm its own stop), and the position — the avg_entry_prices row and the profile_symbols binding — is re-pointed to a target profile on the same account. The target is then reconfigured, which is what makes it tick the inherited symbol at all and seeds its strategy state from the moved cost basis, so it arms its own correctly-hashed stop on the next tick instead of believing itself flat and re-entering on top of the position.

The API has no Binance client, so it may only guard and enqueue; the dispose-profile worker job owns the teardown (disable+unsubscribe → cancel → handoff → re-verify DB and the exchange → wipe Redis → delete the row) and retries until the exchange is provably clear. See docs/architecture/worker-pipeline.md.

Deleting an account always 409s while any child profile holds live exposure, with no escape hatch: the cascade cannot cancel anything on Binance, so a forced account delete is exactly the abandonment above, one level up. Dispose of each profile first.

Orders are the exception, and they behave differently for the two deletes — because an order is ACCOUNT-domain, not profile-domain: its Binance id is unique per symbol not per account (so an account-wide sweep keys on symbol + orderId), the user-data stream that reconciles it is per account, and it keeps resting on the exchange whether or not the strategy that placed it still exists.

• Deleting a profile DETACHES its orders (orders.profile_id → NULL, ON DELETE SET NULL). The order is still real money on Binance, so its ledger row must OUTLIVE the strategy — cascading it away would destroy the only record of a live exchange order. A detached row is reconciled by the detached-orders-reconcile cron (and, while the account still has an active profile, by the user-data stream via fillAdopter.reconcileDetachedFill), and closed when the exchange says the order has left the book.
• Deleting an account CASCADES its orders (ON DELETE CASCADE). The key pair goes with the account, so nothing can ever query, cancel, or reconcile those orders again. There is no one left to keep the row for.

Exposure is counted so that a detached order still blocks the delete:

• Profile: projections.countOpenExposure(ProfileScope).
• Account: projections.countAccountOpenExposure(AccountScope) — counted off orders.account_id directly, not through a profiles join. A join would silently drop every DETACHED row, which is the exact exposure this guard exists to refuse: an account whose last profile was deleted while holding a resting order would otherwise report zero and be deletable. Positions still join through profiles (a position IS strategy state and dies with the profile). Both use the shared isHeldPosition predicate, so the two guards cannot drift.

orders_one_live_per_intent (profile_id, symbol, intent) WHERE closed_at IS NULL is unaffected by the nullable profile_id: Postgres treats NULLs as distinct in a unique index, so a detached row never blocks a new live slot.

`toAccountScope`: minting an AccountScope from a proven ProfileScope Copy Section

Five orders functions moved to AccountScope — findByBinanceOrderId, closeByBinanceOrderId, markFilledByBinanceOrderId, stampRealizedPnl, reapWithReason — because all five seek by (account, binance_order_id) and a detached row is reachable ONLY that way. Narrowing them to the profile would make a detached-but-still-resting order unreconcilable.

Callers that already hold a ProfileScope (the worker's per-profile persistence bindings) must not re-run scopeAccount to reach them: the chain accounts.owner_id = operatorId AND profiles.account_id = accountId AND profiles.id = profileId was already proven, and the account half of it is a strict prefix. toAccountScope(ProfileScope): AccountScope (packages/db/src/repo/_scoped.ts) is the ONE sanctioned way to make that narrowing — it is not a cast, it is the statement that a proven profile chain contains a proven account chain. No other path may mint an AccountScope without scopeAccount.

The account delete enumerates its profiles before the cascade (which removes the rows), then, after the delete commits, wipes each profile's Redis keys and enqueues one unsubscribe-profile job per profile. That cleanup runs under Promise.allSettled and only logs on failure: the destructive write already succeeded, and a 500 would read as "the account is still there". A missed unsubscribe self-heals — the worker re-reads DB truth and a deleted row maps to teardown.

Per-symbol reserve (always-hold floor) Copy Chapter

A reserve is a per-(profile, symbol) base-asset quantity the bot must never sell below — the operator's "always hold 50 ADA, keep trading on top". It is stored on profile_symbols.reserve_base_quantity (decimal-string, null = none) — account/capital data, not strategy config — so it is enforced strategy-agnostically and works for every strategy with no per-plugin code.

The bot trades only the surplus above the floor. Enforcement is a single pure helper, reserveAdjustedBalance(free, locked, reserve) (apps/worker/src/lib/reserve.ts), which drains the reserve from free first, then locked, applied everywhere the worker reads the base-asset wallet balance:

• Boot adoption (runHeldQuantityReconciliation): the wallet is reserve-adjusted before position adoption, so a fully-reserved holding reconciles to a flat position (the bot opens a fresh trade on top) and a partially-reserved holding adopts only the surplus. Without this, the boot reconciler would claim the whole wallet — including the reserve — as the bot's position, deadlocking it (a priced position it can neither trade on top of nor sell).
• Per-tick sell sizing (buildTickInput): the bot-visible base balance is reduced by the reserve, so the pure strategy sees a wallet holding only the surplus and naturally never sizes a sell into the floor. The strategy never learns the reserve exists; golden replays are byte-identical.
• The manual entry-price write (handleApplyAvgEntryPrice): the held quantity it stamps is the surplus, so an operator correcting a cost basis does not re-claim the reserve as part of the position.
• The backtest runner: the same drain over simulated balances, so a backtested reserve behaves as the live one does.
• The post-reconfigure reconcile (reconcileSymbolsAfterReconfigure): the mid-run mirror of the boot sweep, sharing its resolveWalletFields helper rather than deriving the balances itself, so the same profile reconciles to the same held quantity whichever door it came through. This is the door an operator walks through right after setting a reserve; without the drain the bot adopts the ringfenced coins as a position and sells them.

One balance is deliberately not adjusted: unreservedWalletTotal, the raw pre-reserve total threaded alongside the adjusted legs on the boot and reconfigure paths. Only the dust value bounds read it. A reserve is the operator saying which of their own coins the strategy may not sell; minNotional is the exchange saying what it will accept. Only the second can make a holding worthless, so judging the post-reserve surplus would delete a real cost basis over a number the operator can undo by lowering the reserve.

The API (PUT /profiles/:id/symbols/:symbol/reserve) rejects a reserve larger than the live base-asset holding (422), read from the worker-maintained account-info Redis snapshot (with an avg_entry_prices ledger fallback) — never a live Binance call from the API process, preserving the keys-only-in-worker boundary. The reserve is a sell floor only; buys proceed per the strategy. Caveat: as with the wallet reconciler, one profile binding two symbols over the same base applies each symbol's own reserve to the shared balance — rare, and the floor only ever errs toward holding more, never less.

File: docs/architecture/auth.md Copy Chapter

Auth

Canonical references for the auth surface — the in-repo files are the source of truth, this page summarises the constraints those files encode.

• apps/api/src/auth.ts — Better Auth factory: email + password only, argon2id, no SMTP, no 2FA, login throttle 60s / 5, session cookie 24h with sliding refresh < 1h idle.
• apps/api/src/routes/auth.ts — onboarding gate (single master account), sign-up / session / change-password handlers.
• apps/api/src/routes/account-settings.ts — GET/PATCH /account/settings for the master user's account-global display preferences. Today this carries timezone (a validated IANA zone, default UTC) the web UI applies to every rendered timestamp; the value persists on users.timezone.
• packages/db/migrations/0007_better_auth.sql — Better Auth tables.
• packages/db/src/schema/better-auth.ts — drizzle schema for the same.
• apps/api/src/middleware/cors.ts / apps/api/src/routes/ws.ts — the WEB_ORIGIN allowlist (a comma-separated list of exact scheme://host:port origins) gates CORS, Better Auth trustedOrigins (CSRF), and the WebSocket upgrade. Every origin is matched exactly; a * wildcard is rejected at env-parse time because credentialed CORS forbids it. Add a LAN origin to reach the dev server from another device.

Threat model Copy Chapter

The system ships without encryption-at-rest. Binance API keys, notifier secrets, and AI-assist provider credentials (ai_provider_config — the Anthropic Console key / OAuth token and any OpenAI-compatible endpoint key, a spend-capable credential) sit plaintext in Postgres; the mitigation is operator-side and twofold: (1) create the Binance API key without the Enable Withdrawals permission — the bot only reads market data and places spot orders, so a leaked key cannot move funds off the exchange — and (2) IP-allowlist the key at the Binance console so a stolen DB dump cannot place orders without also matching the operator's egress IP. The create wizard and the API-key replace form surface both via the shared ApiKeyGuidance component. The REST client also attaches X-MBX-APIKEY only to the calls it signs, so unsigned market-data reads carry no credential past any proxy or log between the worker and Binance. Step-up / TOTP was deliberately dropped because the system runs single-account on a self-hosted VM — adding a second factor without an email channel for recovery would lock the operator out on phone loss with no fallback.

action_logs rows (msg + ctx) are client-readable at every level, so worker writers MUST keep secrets — API keys, signed URLs, raw credentials — out of action_logs msg/ctx. Three surfaces read them: GET /profiles/:id/action-logs (the activity feed, warn+error only), GET /profiles/:id/logs and its /logs/export (any level, no default level narrowing), and GET /profiles/:id/tick-trace, which returns the raw Redis audit entries verbatim. Deep capture widens this further: while armed, the whole audit payload of every tick is copied into ctx unredacted. Nothing filters these — the rule is that nothing credential-equivalent enters an audit payload or an action_logs row in the first place, because a debug view whose job is showing what the tick saw cannot also be the place a secret is scrubbed.

The process log stream is scrubbed; action_logs is not. Because the credentials above sit plaintext in Postgres, the statement that writes one binds it as a parameter — and drizzle raises a failed statement as an error carrying those bind values three times over: the params array, the message its template inlines them into, and the stack that opens with that message. A logger's default error serializer copies all three, so one failed api_keys insert would write a live Binance secret into the process log, where it outlives the request, the process, and any key rotation. Both loggers (apps/api/src/middleware/logger.ts, apps/worker/src/boot/builders/primitives.ts) therefore run pino's serializer and then scrubDrizzleParams from @app/core/logger, which replaces the bind list wherever it appears — at any nesting depth, through aggregateErrors, and in the chained stack a wrapped cause contributes.

Be precise about what that buys, because it is easy to over-read. It is a backstop for one known error shape, the driver's, not a general redactor: it recognises a params value by its sibling query, and it knows where drizzle puts the bind list in a message. The same known-shape backstop also covers exhausted worker jobs: apps/worker/src/queues/queue-set.ts scrubs the DLQ errorMessage and optional stack before the DLQ Queue.add persists the entry, while apps/worker/src/queues/dlq-watcher.ts and apps/worker/src/boot/builders/notifiers.ts publish and notify from that already-scrubbed errorMessage; this does not make the DLQ path a general payload redactor. It does nothing for a credential a caller passes to a log line itself, and it is deliberately not extended to cover that — a scrubber broad enough to catch arbitrary shapes would be a filter nobody could reason about. The action_logs rule above therefore still governs everything else, unchanged and unweakened: nothing credential-equivalent may enter an audit payload or an action_logs row in the first place. Before adding a route that reads or writes a credential, assume the log stream protects you from the driver's error and from nothing else.

Client-IP trust boundary. The login rate-limiter (apps/api/src/middleware/login-rate-limit.ts) and the audit trail (apps/api/src/middleware/audit.ts) derive the client IP from the rightmost X-Forwarded-For hop via the shared clientIp helper, falling back to X-Real-IP then the literal 'unknown'. This trusts exactly one reverse proxy that appends the real client address to the right of the chain (the bundled nginx $proxy_add_x_forwarded_for config does this). The API therefore MUST run behind exactly one such proxy: expose it directly and the leftmost-to-rightmost chain is fully client-controlled, re-opening per-IP throttle bypass and audit-IP spoofing; front it with two or more proxies and the rightmost hop is an internal proxy, collapsing all clients into one bucket. A request with no forwarded headers records 'unknown' (not SQL NULL) in audit_logs.ip.

Live demo (public, no-login, testnet deployment) Copy Chapter

LIVE_DEMO (env, read by both the api and the worker; parsed strictly — only 1/true enable it, default off) turns a separate deployment into a public sandbox. It is never the operator's real instance, which always requires login.

• No login. sessionResolver (apps/api/src/middleware/auth.ts) injects the boot-resolved sole operator id (repo.users.findSingleId, resolved once in di.ts boot) for every request with no Better Auth session. A real session still wins. Zero-user cold start injects nothing, so /onboarding still works. The login screen never appears (it is only shown reactively by the 401 interceptor).
• Boot refuses a live key. assertLiveDemoInvariant (apps/api/src/di.ts; mirrored in the worker boot) throws when the flag is on and any account is binance_mode='live'. The box can only ever hold testnet keys, so "no sensitive info exposed" holds by construction.
• Locked routes. requireNotDemo (apps/api/src/middleware/require-not-demo.ts, reads di.env.LIVE_DEMO at request time) returns 403 for API-key management; backup, backup-config, and restore; AI-provider management and testing; notifier configuration and testing; ops-notify; account creation; password and session changes; retention changes; and starting a diagnosis run. Interactive trading remains available against Binance testnet. apps/api/__tests__/routes/live-demo-guard-topology.test.ts checks the exact mounted guard set and independently detects an unguarded sensitive route. apps/api/__tests__/routes/live-demo-guard.test.ts preserves request-level 403 samples.
• Notifier suppression (worker). Under the flag both notifier fan-outs (createNotifyEvent, createAccountNotifyEvent) are total no-ops — no dispatch, DB gate unread — so a seed snapshot's real webhooks can never leak.
• Web. The onboarding-status response carries demoMode; the SPA renders a persistent "Live demo" banner and hides the entry points to the now-403 routes (Settings, backup, account settings).

The ops half (testnet golden-snapshot seeding, the separate deployment, the nightly reset that re-restores the snapshot directly rather than via the locked /restore route, and public-endpoint rate limiting) is operator-side. Compose override: deploy/compose/docker-compose.demo.yml.

File: docs/architecture/database.md Copy Chapter

Database

Canonical references for the database surface — the in-repo files are the source of truth, this page summarises the constraints those files encode.

• packages/db/migrations/ — hand-authored, numbered NNNN_*.sql files applied in order by migrate({ connectionString }). They are hand-written, not generated: the drizzle journal is empty so db:generate emits a full snapshot rather than incremental ALTERs and is therefore not used (drizzle-kit is kept for schema typing and db:check). Applied files are tracked by SHA-256 checksum in _app_migrations, so re-runs skip already-applied files. migrate() holds a session advisory lock while applying, so concurrent callers serialize and the loser skips the already-applied files instead of racing on non-transactional create type DDL. This is a migration-time DB lock, not a runtime coordination lock, so it does not breach the no-distributed-locks invariant. CI runs the packages/db isolation/projection suites against a real Postgres via DATABASE_TEST_URL in the integration job; without that variable they skip. DATABASE_TEST_URL must name a database whose name ends with _test (e.g. binance_trading_bot_test): the integration and isolation harnesses TRUNCATE / delete rows on it, and assertTestDatabaseUrl refuses to run against any non-_test target so a value pointed at the live binance_trading_bot cannot wipe real data.
• A migration is immutable once it has shipped. _app_migrations keys on the file name and stores the body's SHA-256, so renaming, renumbering, deleting, or editing an applied file — a comment is enough — makes the runner throw already applied with a different checksum. Refusing to mutate history. on every database that already holds the old value. It throws at that file, before any later migration runs, and the checksum map is read once up front, so a repair migration cannot execute itself out of the hole: recovery means restoring the original bytes or hand-editing the ledger. To change what a shipped migration did, write a new one. CI cannot see this on its own, because every suite migrates a fresh database where a mutated file looks correct; scripts/ci/no-mutated-applied-migration.sh supplies the missing oracle by pinning each file's digest in packages/db/migrations/checksums.json. Adding a migration means adding its line to that manifest.
• packages/db/src/schema/ — drizzle schema definitions, one file per logical table group.
• packages/db/src/repo/ — typed repository wrappers over a two-tier scope. Account-scoped functions take an AccountScope (produced only by scopeAccount); profile-scoped functions take a ProfileScope (produced only by scopeProfile). Each scope runs the single ownership check for its tier, so a scoped query cannot be reached without ownership having been proven.

Account isolation policy Copy Chapter

The data model is one operator (users) → many accounts → many profiles. An account is a first-class row (accounts, owner_id → users.id) holding one Binance API key pair, one binance_mode, and one user-data stream; profiles.account_id → accounts.id. Ownership is a two-hop chain — accounts.owner_id = operatorId and profiles.account_id = accountId — resolved in a single query by scopeProfile(db, operatorId, accountId, profileId) (and scopeAccount(db, operatorId, accountId) for account-level surfaces). UserId (the operator) and AccountId are distinct branded types, so misplacing one for the other is a compile error. The typed repository layer is the single enforcement point; app code resolves a scope via profileRepo / accountRepo (or the *FromScope variants) and ownership is checked exactly once per scope. There is no DB-side RLS — the cross-account integration tests in apps/api/__tests__/cross-account.test.ts and packages/db/__tests__/isolation/*.test.ts verify the policy in CI, and packages/db/__tests__/repo/ast-check.test.ts statically enforces the scope-first contract.

See also: account-isolation.md.

`override_actions`: `result` vs `outcome` Copy Chapter

override_actions backs two unrelated operator flows, and each owns its own nullable jsonb column. They are not interchangeable:

outcome is stamped by EVERY terminal transition, so "the row is closed out" and "the row carries an outcome" are one fact — a row marked done with no outcome cannot tell a filled force-sell apart from one the exchange refused, and reads on the symbol page exactly like a success. The two columns stay separate for the same reason: sharing one would make null mean both "still pending" and "settled, but the payload is not an outcome", which no reader can disambiguate without guessing at the shape.

Migration 0072 adds the outcome column plus the two indexes these reads need — a (profile_id, symbol, created_at desc) index for "the newest override for this symbol, settled or not", and a partial index on the pending symbol-scoped rows for the stranded-row sweep.

All four terminal paths funnel through one private consume() in packages/db/src/repo/override-actions.ts, whose outcome argument is required, so a new "mark it done" writer cannot reintroduce the outcome-less row. The stranded-row sweep is account-tier (reapExpiredForAccount, on AccountRepo): two statements per account per sweep — one per stranding branch, see below — rather than a scope-resolve plus an UPDATE for every active profile.

`picked_up_at`: a write-once record, not a lease Copy Section

Migration 0075 adds a third nullable column, and it is the odd one out: nothing is gated on it, and nothing ever clears it.

(It is of course in the WHERE of both sweep statements — "gates" means no path branches on it the way the cancel route branches on a claim, and the dispatch gate on the claim's own CAS reply.)

The "cleared by" column is why this is a new column and not a reuse of processing_at. A claim is a lease: it exists in order to be handed back. Two paths null it — releaseClaim, when a side-effect failed and the next tick should retry, and reapStaleProcessing, when a worker died holding a claim. A lease therefore cannot carry a fact that must OUTLIVE the crash: the crash is what summons the stale-claim reaper, and the reaper clears exactly the evidence the sweep would need to read. picked_up_at is written once and cleared by nothing, so it survives.

This is why the two columns stayed separate once the tick DID become a claiming consumer of overrides. The claim means "work is in flight now", which is inherently revocable; the breadcrumb means "work was once in flight", which must never be revoked. Merging them would put the durable fact back under a reaper's control.

The two also differ in span, which is what keeps a claim from swallowing cancellation. processing_at is read as a guard — deletePendingForSymbol skips a claimed row — so the tick holds it only across the window where a dispatch is genuinely in flight, and settleOverride releases it before re-arming the Redis key. An operator cancel arriving inside that window is answered 409, never silently dropped; outside it the row is deletable again, and the compensating re-arm still infers "the operator revoked this" from that cancel having deleted the row. consumed_at is terminal and would settle the override the tick is still holding.

Both columns are also read at ARM time. record settles the row a new override replaces superseded, in one transaction with the insert, because the operator's Redis key is overwritten blindly and only the newest override can run — but it settles only rows where BOTH are null. A claimed row would disappear from findActiveForSymbol and take the cancel route's 409 with it; a breadcrumbed one is destined for the sweep's unknown, the only outcome that notifies, and terminal writes are immutable, so an early superseded would suppress it permanently. Same asymmetry as everywhere else here: the lease and the breadcrumb both mean "someone else owns this row's ending".

No index. The sweep's predicate is unchanged and its two branches are disjoint halves of the same pending set already covered by override_actions_pending_symbol_idx; a second index on a two-valued column would earn nothing and cost every override write. What the two branches mean for the operator is in worker-pipeline.md.

`orders`: account-owned, profile-referencing Copy Chapter

An order is ACCOUNT-domain: its Binance id is unique per account, the user-data stream that reconciles it is per account, and it keeps resting on the exchange whether or not the strategy that placed it still exists. Migration 0073 moves the table onto that footing:

The backfill is total (profile_id was NOT NULL with an FK before, so every row joins to a profile and every profile to an account), which is what lets SET NOT NULL on account_id be unconditional.

New index orders_account_binance_order_id (account_id, binance_order_id): the seek every reconciliation path now makes (user-data stream, orphan sweep, adopt route, detached-orders-reconcile), and the only way a detached row is reachable at all.

The partial unique index orders_one_live_per_intent (profile_id, symbol, intent) WHERE closed_at IS NULL needs no change: Postgres treats NULLs as distinct in a unique index, so no two detached rows ever conflict and a detached row never blocks a new live slot for the (recreated) profile.

A recovery row — an order that IS live on Binance but whose normal bookkeeping did not land — is written by orders.insertTracking under a reserved intent, `${intent}:untracked:${binanceOrderId}`. intent is an open, strategy-owned string (no CHECK since 0026), and the reservation is what keeps the row out of the strategy's live slot: that slot is very often ALREADY HELD by the still-resting previous order — which is the single most likely reason the normal write failed in the first place — so an insert under the strategy's own intent would conflict on the partial unique index and be silently swallowed, leaving the live order with zero local trace. The row stays fully visible where it matters: the orphan sweep and the exposure guard read account_id / closed_at (intent-blind), and the fill adopter seeks by (account_id, binance_order_id).

`trade_archive.missing_cost_basis`: an under-count must say so Copy Chapter

profit is cost-basis matched, so a SELL whose realized_pnl is NULL contributes nothing rather than booking its proceeds as a zero-cost gain. That arithmetic is correct, and it is also unreadable once written: profit = 0 renders as +0.00, and the operator reads a real trade as flat. The count was already computed at archive time and only logged, which put the whole signal outside the row it describes.

Migration 0080 adds missing_cost_basis integer NOT NULL DEFAULT 0: how many of the cycle's SELLs had no cost basis. A positive value means the row's profit and profit_percent — and the net figure the API derives as profit − fees_quote, which is not a stored column — are a conservative under-count, and the API carries it so the archive page shows "P/L unavailable" instead of a number nobody measured.

The bought and sold totals under-count on exactly the same rows, so they are not a safe fallback: total_buy_quote is Σ cost_basis_quote, which an un-costed SELL contributes nothing to, and total_sell_quote is derived as total_buy_quote + profit so that profit = sell − buy holds exactly. A fully un-costed cycle therefore renders 0 in both money columns despite real coins changing hands, and the period rollups above the table count it as zero. The archive page's inline note says so rather than presenting those figures as measured.

The default is 0, which is not a claim about existing rows — it is the assumption the UI already made before the column existed. Backfilling it would mean re-deriving cost bases that no longer exist, which is precisely the fabrication the NULL was protecting against.

`trade_archive.archived_at`: paginate on the microsecond token Copy Chapter

The archive page pages by keyset on (archived_at, id), and archived_at is timestamptz — microseconds. The cursor it emitted was that row's timestamp as the node-postgres driver hands it back: a JavaScript Date, which is milliseconds. A row at .123200 is strictly older than a page boundary at .123456, yet satisfies neither archived_at < .123000 nor archived_at = .123000, so the walk skipped it on every later page and nothing reported the loss — a silent failure in the operator's own trade history. Two archives in one millisecond is ordinary: a recovery backfill writes a cycle per iteration.

listForProfilePaginated therefore returns each row with a cursorToken — to_char(archived_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"'), the full microsecond value as an ISO string — and the route emits ${cursorToken}__${id} and binds the timestamp half straight back as a string, cast to timestamptz in the predicate so it stays a direct column comparison rather than a string compare. Never re-parse that half through new Date: that is the truncation the token exists to avoid. This is the same treatment audit_logs, action_logs and backtest_runs already use; trade_archive was the paginated timestamp column that had not received it. The column keeps its precision — only the cursor changed. Two tests pin the two halves, with two rows differing only below the millisecond each: packages/db/__tests__/isolation/trade-archive.test.ts pins the predicate, binding a full-precision token straight back, and apps/api/__tests__/routes/archive.test.ts pins the emit side — that nextCursor carries six fractional digits at all, and that the sibling row is reachable on the next page. The boundary schema also rejects the two ISO instants Postgres cannot bind: there is no year zero in AD/BC notation, so 0000-… comes back as SQLSTATE 22008, and a fractional second long enough to overrun the datetime parser's fixed work buffer is refused outright rather than rounded away — each an unhandled 500 on a route whose declared failures are 422 and 503. z.iso.datetime() is calendar-aware (it rejects month 13, day 32, 2023-02-29, 24:00, :60) but leaves exactly those two gaps, so isBindableTimestamp in @app/contracts closes them with a length cap and a year-range test — a positive range assertion, not a 0000- blocklist, because a blocklist only refuses the spellings it was told about. Every cursor reader shares it through one compositeCursor factory (apps/api/src/lib/cursor.ts), parameterised by separator and by whether a bare timestamp is honoured, so the archive, audit-log and backtest-run readers cannot drift apart; the action-log cursor's own regex still owns its microsecond precision and defers the calendar to the same predicate.

The keyset predicate is a row comparison, (archived_at, id) < ($ts::timestamptz, $id::uuid), not the equivalent a < x OR (a = x AND b < y) pair. Both return the same rows, but only the row comparison becomes a btree start condition: with the OR form the boundary stays a per-row filter, so every page walked the profile's archive from its newest row and discarded everything above the cursor, and page cost scaled with the archive's total size rather than with limit. trade_archive_profile_archived_id (migration 0086, (profile_id, archived_at desc, id desc)) supplies the matching order. Measured on 5,000 rows with the cursor 4,000 deep: before, a Seq Scan feeding a top-N heapsort, 4,003 rows removed by filter, 92 shared buffer hits, 2.374 ms; after, an Index Scan with both cursor columns in the Index Cond, 0 rows removed by filter, 4 shared buffer hits, 0.035 ms. packages/db/__tests__/isolation/trade-archive-plan-shape.test.ts pins all three properties — no Sort node, an Index Cond naming the index and both cursor columns, and zero rows removed by filter — because a fix that only added the index would still leave the boundary as a filter.

`backfill_attempts.symbol_unavailable`: a terminal reason the counts cannot express Copy Chapter

The other reasons a coin lands in the "nothing to recover" note are derived from the reconstruct drop counts, and all of them assume the trade history was read. A delisted coin never gets that far: the backfill handler needs the symbol's base/quote assets from the exchange-info cache, and without them it cannot value a single fill.

Migration 0081 adds symbol_unavailable boolean NOT NULL DEFAULT false. The handler stamps it when the symbol is absent from a primed cache — absence alone is not evidence, because a cold cache looks identical, so the keyspace is probed for the account's mode first and the job is retried when it is empty. The flag outranks the count-derived reasons on the API projection: with round_trips, skipped_orphan_sells and dropped_overshoot all 0, the count-derived reason would read open-or-pre-history, which names the wrong cause and points the operator at a window that does not exist.

Like 0080's default, false is not a claim about existing rows: it is the state every row already implied before the column existed, and a marker re-opens by itself once a later fill makes it stale.

`condition_states`: a level store beside the edge stream Copy Chapter

action_logs records edges — that something changed. Operators ask level questions: what is wrong now, and since when. The two are not interchangeable, and writing only on change is what exposes the gap: a symbol stuck on one reason for three weeks has exactly one transition row, written three weeks ago. Once that row is past the retention horizon the query returns nothing, so the log viewer is emptiest for the most-stuck symbol, which is the worst possible failure mode for a diagnostic.

Migration 0077 therefore stores the level as a level. condition_states is keyed (profile_id, condition, symbol), carries the reason as code plus the producer's own detail payload, and stamps since when the reason began. since is the column that makes aggressive action_logs pruning safe: durations stay exact after the opening edge has been swept. Size is bounded by open conditions × symbols rather than by time, because a row exists only while its condition is open and resolving deletes it.

It is a plain table rather than more action_logs rows for a reason that is structural, not stylistic. action_logs is a TimescaleDB hypertable partitioned on time, and a unique index on a hypertable must contain every partitioning column — so a key of (profile_id, condition, symbol) cannot exist there at all, and adding time to it makes every write a new row rather than a replaced one. ON CONFLICT DO UPDATE on the state key is simply unavailable. The read is also unindexable in practice: "current condition per subject" over the log is a DISTINCT ON over a jsonb expression key that none of the three action_logs indexes can serve, where here it is a primary-key lookup.

symbol is an empty-string sentinel rather than NULL when the condition is about the profile itself, because Postgres forbids nullable columns in a primary key and the alternative — two partial unique indexes — would force two upsert paths for one write. PROFILE_SUBJECT is exported from @app/db so no reader has to spell the sentinel itself. condition is text rather than an enum so adding a producer needs no migration; the closed set lives in packages/contracts/src/condition.ts, where readers already validate it.

Both writes go through one recordCondition, which writes nothing when the code is unchanged, so the per-tick hot path stays free. 0082 adds a nullable change_key for the case where the reason holds but the threshold it names has moved: comparing code alone dropped that write, leaving the stored detail advertising a number that is no longer the live gate.

Dashboard read-through caches Copy Chapter

The two dashboard projections cache their composed payload in Redis to absorb the SPA's 5s poll without re-running the per-profile × per-symbol Postgres + Redis fan-in on every request:

• getAggregateForAccount — the per-account cross-profile home rollup. Key dashboardAggregateCacheKey(accountId) (tenant:<accountId>:dashboard-aggregate:cache), TTL DASHBOARD_AGGREGATE_TTL_S = 5.
• getProfileDashboard — the per-profile view. Key profileKey(scope, 'dashboardCache'), TTL PROFILE_DASHBOARD_TTL_S = 5.

Because reads are cached, a write must drop the affected keys or the UI replays a stale payload until the TTL expires (the "action is slow to refresh" lag). The bustDashboardCache middleware (apps/api/src/middleware/) is mounted app-wide and, after any successful (2xx) non-GET request by an authenticated user, reads the {accountId} from the account-scoped path and deletes that account's aggregate key plus the per-profile key when the path also carries a {profileId}. The SPA's on-success refetch then recomputes immediately. Invalidation is best-effort via invalidateDashboardCaches — a Redis failure is swallowed so it never turns a successful write into a 5xx; the read just waits out the TTL. The cache holds composed DTOs only; the source of truth stays in Postgres, and ownership is re-proven on the recompute path, so a stale or dropped cache is never an isolation concern.

Hypertables: the root heap Copy Chapter

action_logs and candles are TimescaleDB hypertables (0005_hypertables.sql). Their rows live in chunks; the parent relation — the root heap — is meant to stay empty. TimescaleDB's planner expands a hypertable to its chunks and leaves the parent out of the plan, so a row that does end up in the root heap is invisible to every statement naming the hypertable: select, update and delete alike, with or without only. Chunk-drop retention never reaches it either, so a stranded row outlives every retention horizon indefinitely.

alter table ... set not null is the exception. It does not go through the planner, it scans the heap directly. A migration that backfills a column and then constrains it is therefore reading two different sets of rows, and fails on exactly the rows nothing else can see — the backfill reports every row updated and the constraint still reports a null. This is what aborted 0076 after deployment: action_logs had 17 rows stranded in its root heap, and all three Job retries failed identically. 0075a_action_logs_root_heap_drain.sql drains them back through the hypertable (delete from only ... feeding an insert, so routing files them into a chunk), which also puts them back under the retention horizon.

How the rows got there was not established. The one documented way to reach the root heap is timescaledb.restoring, the mode timescaledb_pre_restore() sets so a dump can be reloaded without routing; an insert made while it is on lands in the parent. Whether that is what happened here is unproven — the Postgres logs for the window had rotated away.

Any migration adding a constraint to a hypertable column must drain the root heap first. When a backfill claims success and the constraint still fails, compare from <hypertable> against from only <hypertable> before theorising about concurrent writers.

One measured subtlety: the parent is excluded from the plan only once the hypertable owns at least one chunk, so a backfill on a chunkless hypertable does reach the root heap. No regression test covers this — the insert has to be made under timescaledb.restoring, which survives in the parent through 2.27.1 and is discarded from 2.28.0, so the fixture cannot be built on the deployed 2.29.1 image.

Pool checkout deadlines Copy Chapter

Every pool is created by createPool (packages/db/src/pool.ts) with connectionTimeoutMillis: POOL_CHECKOUT_TIMEOUT_MS — 5 seconds, for all three kinds. Left unset, pg-pool queues a checkout with no timer at all: once max connections are held, every later checkout on every route waits indefinitely, with no error, no log line, and no bound of its own. The api pool is 10 connections, so one route holding four of them per request needed only three concurrent requests to take the whole process dark while its health check kept answering.

The one option arms both of pg-pool's deadlines, which is why the value is seconds rather than the few hundred milliseconds a queue wait alone would deserve — it also has to cover a cold connect to a healthy database:

• queue wait — the pool was full and nothing came free in time. Rejects with Error('timeout exceeded when trying to connect').
• cold connect — a brand-new connection did not finish its TCP + startup handshake in time. Rejects with Error('Connection terminated due to connection timeout'), carrying the driver's Connection terminated unexpectedly on cause.

Neither error carries a SQLSTATE or an error class, so poolCheckoutTimeoutKind classifies them by exact message equality (walking the cause chain, because drizzle wraps what the driver threw). Exact rather than substring on purpose: a reworded pg-pool release then fails closed — a saturated pool degrades from 503 back to 500, which is wrong but not a lie — and takes packages/db/__tests__/pool-checkout-timeout.test.ts red with it, since that suite drives a real pg.Pool into both paths against a net blackhole rather than asserting hand-written fixtures. Substring matching would instead relabel any unrelated error whose message quotes the phrase as backpressure.

One value serves all three kinds because only api ever queues: worker fans out per job against 25 connections, admin is max: 2 with no concurrent fan-out, and migrate() opens a bare pg.Client so migrations never touch a pool. It is a constant rather than an environment variable — the failure it prevents is unbounded queueing, and an operator who can set it can also set it back to a hang.

The api error handler maps a classified checkout timeout, and a statement cancelled by withStatementTimeout (SQLSTATE 57014), to 503 SERVICE_UNAVAILABLE, logged at warn with the request path. They are load events, not defects: an unhandled error line for a saturated pool reads as a bug and sends whoever is on call looking for one.

Each of the three gets its own message and its own log name, because they take different remedies and the log line is what an operator acts on:

Collapsing the first two into "connections exhausted" would be wrong for the second: pg-pool only dials a new connection when the pool has room, so a cold-connect timeout means the pool was not full, and pointing the operator at the pool size aims more concurrent attempts at a server already failing to answer.

The two bounds are complementary and a route needs both. GET /profiles/{id}/trade-archive is the worked example: its four reads used to run as a Promise.all, taking a pooled connection each, and now run in sequence inside one withStatementTimeout(di.db, 5_000, …) transaction whose ownership scope is minted on the same connection. One request therefore holds one connection. GET /profiles/{id}/discovery and GET /account/health now follow the same shape, and the health bar is the reason it matters most: it summed realised P/L per profile, so its checkout burst grew with the profile count and an operator with a handful of profiles could empty the pool on one poll of a bar that polls. One grouped rollupRealizedByProfileForAccount read replaces that fan-out, left-joined from profiles so a profile that closed nothing still reports a zero in its own currency. Redis reads stay outside these transactions and concurrent — a different pool with a different failure mode, and holding them outside costs nothing. The transaction issues up to five statements that can stall on a view=full request — the ownership join, then the four reads — so the worst case hold is 25s, five times the per-statement budget. A view=rollup request, which is what every polling surface asks for, issues two. That deliberately exceeds the 5s checkout deadline: under sustained archive concurrency a waiter can be rejected while the holder is still legitimately working. The budget bounds a pathological stall, not the healthy case (roughly a tenth of a second per read), and it is not sized down to make the arithmetic tidier because there is no production timing for listForProfileInRange against a large archive. It no longer doubles as a size limit on the archive: the paginated read is index-served and costs limit rows, so listForProfileInRange — unpaginated by design, since the rollup is period-scoped rather than page-scoped — is the read this cap now exists for. apps/api/__tests__/routes/archive.test.ts asserts both properties from the pool itself — peak concurrent checkouts during one request, and that a set_config('statement_timeout', …) was actually issued on the acquired connection — because the response body cannot show either.

One consequence for alerting, and one thing the metric's name hides. DBPoolStarved fires on pg_pool_waiting > 0 held for 5m, but that series is sampled by the worker, over the worker pool (apps/worker/src/boot/runtime-gauges.ts), and nowhere else — the api pool has never had a gauge, so api-side starvation was invisible here long before this change. For the pool the rule does watch, the deadline does weaken it: the gauge is an instantaneous waitingCount, so it never accumulated, it stayed high, and a waiter ejected after 5s is less likely to hold it non-zero across five consecutive one-minute samples. The rule is deliberately left as-is (any waiter is still the condition, and retuning it without production data would trade sensitivity for noise). The api's equivalent signal is http_requests_total{status="503"}, which an ejected checkout now raises — a different pool in a different process, so a 503 rate with DBPoolStarved silent is a different incident, unless Postgres itself has run out of connection slots and both pools starve together.

One asymmetry this leaves in the OpenAPI document. A pool-checkout 503 is raised by errorHandler before any route-specific logic, so it is now reachable on every route, while only the routes that open a withStatementTimeout transaction of their own declare 503:. That is a known gap in the document, not a claim that the response cannot happen. Filling it in route by route is the wrong repair — 124 createRoute calls each declare their own responses, so enumerating a transport-level condition on every one of them drifts the moment a route is added, and a partially-applied version is worse than none. The repair is a shared response builder every router composes, which is a change to the whole api surface rather than to the pool, so it is tracked separately; the routes that declare 503: today are the ones that raise it from their own statement budget, which is a route-level fact and does belong on the operation.

File: docs/architecture/extensibility.md Copy Chapter

Extensibility

The platform runs one strategy per profile, and a strategy is a plugin. The @app/strategy-core package (packages/strategy/core/) is the canonical type home of the plugin contract; this page is its prose companion. Every type named here links to its definition in that package, and when the two disagree the code wins and this page is the bug.

Trailing Trade is the first packaged strategy, momentum the second. Adding a third is a new package plus one registry entry, never an edit to apps/api or apps/worker (core invariant 1).

Canonical references:

• packages/strategy/core/src/contract.ts — Strategy, Capabilities, TickInput, TickOutput, the state-adapter seams.
• packages/strategy/core/src/decision.ts — the Decision union, OrderIntent, OrderParams.
• packages/strategy/core/src/registry.ts — createRegistry, StrategyRegistry, AnyStrategy.
• packages/contracts/src/operator-actions.ts — the OPERATOR_ACTIONS closed set behind capabilities.operatorActions.

The Strategy interface Copy Chapter

A plugin is one object satisfying Strategy<Config, State, Bundle, Events> (contract.ts). Its members:

previewLevels is guarded by a drift gate wired into golden-fixture replay (assertPreviewTickAgreement in @app/strategy-core, called from replayFixture and trailing-trade's own replay loop). For every decision a tick emits carrying a reason R, the gate gathers the preview's trigger rows with code === R and a price; if there are any, at least one must have currentPrice on its triggerWhen side. This is the weak emitted ⟹ consistent implication only — a reason with no price-bearing trigger row is exempt, and the converse (crossed ⟹ emitted) is never asserted, so a decision gated on a non-price condition (an EMA cross, a technicals signal) never false-fails the replay. A disagreement means the operator's pre-trade view lied about where an order acts.

Rendering the preview (apps/web) Copy Section

The SPA renders a strategy's PreviewModel with one generic component, so a new strategy needs zero web code beyond registering it. Each strategy exposes its pure previewLevels + previewDataNeeds on a ./preview subpath export (packages/strategy/*/package.json), and the web lazy-imports it through a single map (apps/web/src/features/symbol/preview/preview-modules.ts, keyed by strategy name). Because the import is dynamic, each strategy's decimal.js math stays in a code-split chunk — the SPA itself is decimal.js-barred and only formats the decimal-string fields the model returns.

• usePreviewModel loads the module, fetches the candle windows the config declares (its candleInterval decision window plus any previewDataNeeds extras), and runs the pure previewLevels. It is form-independent, so both the config-page aside (fed a react-hook-form useWatch draft) and the live symbol workspace (fed the persisted config) share it.
• StrategyPreviewPanel renders the model as titled Panels of level rows — price/quantity/skip/symbol/weight/drift cells — replacing the former per-strategy ConfigPreview. Mounted at the profile-config, symbol-config, and backtest-configure editors.
• deriveChartLines turns the model's chartLine priced rows into the candle chart's horizontal lines, replacing the former per-strategy chartLines. The PreviewTone → token/ChartLineTone maps live in one place (preview/tone.ts), so the panel and the chart agree on colour (notably the trailing line).
• A strategy that sizes off free cash (momentum's percent-of-account entry) reads balances from PreviewInput.account, which is an AccountSnapshotWire (decimal-string balances the SPA can build); the strategy revives them to Decimal at its own boundary.

scripts/ci/no-missing-preview-export.sh fails CI unless every registered strategy has both the ./preview export and an entry in the web import map.

tick() is pure: no I/O, no wall clock, no randomness beyond the injected Clock and RNG on TickInput. It receives one per-(profile, symbol) state slice and returns the next slice plus a list of Decisions. The worker, not the strategy, performs every side effect those decisions describe.

Capabilities Copy Chapter

Capabilities (contract.ts) is what the worker reads to wire a profile. It declares infrastructure feeds, not an operator-facing descriptor:

operatorActions tokens are validated against OPERATOR_ACTIONS (packages/contracts/src/operator-actions.ts) at the api wire boundary (the OperatorAction z.enum) and by the registry consistency test. An action a strategy does not declare is gated 422 at the api, dropped by the worker, and omitted from the web UI, so an action that would be silently dropped never renders. operatorActions and bundleProviders are both readonly string[] to keep this package free of a @app/contracts dependency; the closed set is enforced at the boundary, not by the type.

TickOutput Copy Chapter

tick() returns { nextState, decisions, logs, metrics, events?, overrideDeferred?, overrideDeclineReason? }. nextState is the per-(profile, symbol) state slice the worker persists; decisions drive the executor; logs / metrics / events are the operator, counter, and typed audit channels.

overrideDeferred is the strategy's answer to a question only it can answer: did I act on the operator override you handed me? The worker removes the override from Redis before calling tick() (at-most-once), so a strategy that quietly ignores one would have the operator's "flatten now" reported as executed while nothing sold. Set overrideDeferred: true and the worker instead re-arms the override key — SET <key> PX <remaining-ttl> NX, restoring the operator's original expiry window, yielding (NX) to any newer override pushed in the gap — and leaves the override_actions row un-consumed, so the SPA keeps showing it as pending, which is the truth. Absent or false consumes the override, the normal outcome.

Only set it for a transient blocker: momentum defers a force-sell while heldQuantity is still null (the boot-time held-qty reconciler pins it within a tick or two), and during candle warm-up. A permanent refusal must NOT defer — an exchange filter rejecting a dust position would re-arm the same doomed override every tick until the TTL expired.

overrideDeclineReason is the strategy's short reason for emitting no order for the override — a symbol filter it could not satisfy, a cooldown, a warm-up. The worker stores it verbatim on the override row so the operator reads the real reason rather than a generic "the strategy declined". Set it on any tick that declines an override, deferred or not.

Order provenance Copy Section

A strategy that emits an order because of an override stamps that override's id on the order: intent.overrideActionId. This is the ONLY thing that lets the worker tie an order's real outcome back to the override the operator is watching. Without it the worker can settle the override only on the strategy's INTENT to act, so an order that was throttled, rejected by Binance, or dropped by the daily-loss breaker still reads as "done". Attribution is by id equality and never by a heuristic — "a BUY got dropped, it was probably the override's" settles the wrong row the moment a tick emits an unrelated order, which is every grid tick.

Outcome, not intent Copy Section

The worker settles an override on what happened to that order, and records the outcome on the row (applied / rejected / unknown / superseded / expired), which the SPA shows. Whether the order may be re-issued is decided by two INDEPENDENT facts on the failed DecisionResult:

• phase — is a retry SAFE? pre-call (never sent) and rejected (Binance parsed it and refused) both prove nothing executed. ambiguous (transport error, HTTP 5xx, an error body with no parsable code) means the order MAY be live, and accepted means it IS.
• retryable — is a retry WORTH IT? A weight throttle or a -1003 clears; a -2010 insufficient-balance does not.

DecisionFailurePhase is an executor-contract concept (the backtest executor reports it too), so it lives here in strategy-core. Deciding WHICH phase a live Binance failure is in does not: only the REST client can see whether Binance's error body actually parsed, so @app/binance computes BinanceApiError.phase at throw time, alongside the precomputed retryable, and the worker's error taxonomy reads it. A downstream re-derivation from status/error-code heuristics would silently flip ambiguous → rejected the day that detail changed — and that flip re-arms an order that may be live on the exchange.

The override is re-armed iff (phase === 'pre-call' || phase === 'rejected') && retryable. An ambiguous failure is never re-armed, whatever retryable says: a retry could place a second live market order. It settles as unknown and notifies the operator, because that is the one outcome only a human can resolve, at the exchange.

The worker refuses to re-arm, and settles the row with the outcome the operator actually sees, when:

• the override's action is not in capabilities.operatorActions — it could never be honoured, so a re-arm would loop to the TTL. Settles rejected ("this strategy does not support this action");
• the remaining window is gone. The TTL is read before tick() and this tick's own latency is charged against it, so a re-arm restores the operator's original deadline rather than extending it on every defer; with nothing left to restore, the row settles rejected with the strategy's own decline reason;
• the tick ALSO placed an order under this override's id — a defer means "I could not act", and re-arming would hand the same override to the next tick, which would place a second order under a different clientOrderId (Binance dedups a clientOrderId only while the first order is still open). One accepted order settles the row applied whatever its siblings did; the worker logs an error and does not trust the flag over the evidence;
• the SET … NX lost its race, i.e. the operator pushed a NEWER override into the key between the consuming DEL and the re-arm. The fresher intent wins, so the stale row settles superseded rather than being left pending forever behind a key it can never own.

If a re-armed override's window lapses while the strategy is still unable to act, the Redis key expires and the row is left pending — nothing executed, so marking it done would be a lie. The dust-snapshot sweep settles such a stranded row as expired once it is older than the outcome window, so the operator is never left watching an override that can never run.

The Decision union Copy Chapter

tick() returns Decisions (decision.ts). The union exposes exactly these variants today:

set-kv / delete-kv are the cross-symbol seam: a strategy trading several symbols on one profile gets one tick() slice per symbol, so it cannot read a sibling's state directly. It publishes facts via these decisions and reads the merged store back on later ticks through TickInput.profileKv — present only when the strategy opts in via capabilities.needsProfileKv. Writes are last-writer-wins per key across concurrent sibling ticks (eventual, like the rest of the contract); a strategy republishes its slice each tick. The store is per-PROFILE, not per-symbol. The union still stays generic: strategy-specific side effects ride on the variants above with namespaced strings, never a new union variant (the CLAUDE.md anti-pattern).

State scope Copy Chapter

The State generic on Strategy<Config, State, Bundle, Events> is the per-(profile, symbol) slice the worker loads, passes to tick(), and persists after each tick. A strategy trading multiple symbols on one profile receives one slice per symbol per tick; there is no profile-wide state object.

flowchart TD
    Tick["Strategy tick"]:::core
    PerSym["TickInput.state — per profile and symbol, symbol_states table"]:::io
    ProfKv["TickInput.profileKv — per profile, profile_kv table, opt-in via needsProfileKv"]:::io
    NoWide["No profile-wide state object"]:::wait

    Tick --> PerSym
    Tick --> ProfKv
    Tick -.-> NoWide

    classDef io fill:#2c3e50,color:#ffffff;
    classDef core fill:#1f6f43,color:#ffffff;
    classDef gate fill:#f0ad4e,color:#2c3e50;
    classDef act fill:#1f6f43,color:#ffffff;
    classDef wait fill:#ecf0f1,color:#2c3e50;

Swapping a profile's strategy Copy Section

symbol_states stamps strategy_version but never the strategy name, so a slice left behind by the outgoing strategy is indistinguishable from one the incoming strategy wrote — the worker's reconcile spine would hand the old body to the new strategy and ask it to migrate from a version it has never shipped. repo.profiles.switchStrategy therefore deletes every symbol_states row for the profile in the same transaction as the profile update. The next tick per symbol re-seeds from strategy.initialState(config); the reconcile spine clears the matching Redis key on that read, so no stale cache survives either.

Concurrency Copy Section

Ticks for different symbols of the same profile may run in parallel. The executor's chainByKey serialises calls per (profileId, symbol) within one process. The durable cross-pod guard is a symbol_states.version optimistic CAS: every state write is WHERE version = expected and bumps it, so a stale writer (a fill racing a tick across pods) matches zero rows — the fill path re-reads and retries, the tick commit skips without clobbering. Order placement is idempotent via deterministic clientOrderId. chainByKey remains the intra-pod serialiser and a fast path; it is not the cross-pod mechanism. Boot reconcilers rely on pre-queue temporal exclusivity at startup, not the lock.

Anti-patterns Copy Section

• Reading state.avgEntryPrice for symbol X from inside the tick of symbol Y. There is no shared state object; the tick of Y sees Y's slice only.
• Adding a new Decision variant for "strategy-specific KV". Cross-symbol KV rides on set-kv / delete-kv under namespaced keys; never add a per-strategy union variant. The union stays generic per the CLAUDE.md note.

State-adapter seams Copy Chapter

One optional capability lets the worker converge per-symbol state without importing a plugin's concrete schema (core invariant 1). A strategy that omits the adapter is simply skipped by the matching worker path.

• position?: PositionStateAdapter<State> (contract.ts). Present when the strategy manages a single long position per (profile, symbol). The worker's boot reconcilers and fill-adopter read readPosition (projecting the generic PositionView) and merge an AdoptedFill through applyFill, so they make Decimal decisions without naming a state-body field. The same adapter clears the position via clearPosition(state, opts?): a phantom-ledger prune calls it with no options; the operator reset-grid action (the worker pipeline handler is historically named reset-grid-trade) cancels the live grid orders (generic order-domain work) and calls clearPosition(state, { resetGridIndex: true }) to also abandon the current grid cycle, again without the worker naming a field. A strategy with no grid index treats resetGridIndex as a no-op. The reset-grid clear is gated on the strategy declaring reset-grid in capabilities.operatorActions (the same gate the API enforces), not on a dedicated adapter slot. Each method returns the next body or null for "no change / not my schema".

Adding a strategy Copy Chapter

The empirical path momentum took:

1. New package. packages/strategy/<name> with npm name @app/strategy-<name>, exporting one object that satisfies Strategy.
2. Declare capabilities. The candleIntervals and feeds the worker must provide, the bundleProviders the strategy reads, and the operatorActions it honors (a subset of OPERATOR_ACTIONS).
3. Register it. Add one registry.register(<strategy>) line to buildStrategyRegistry in @app/strategy-registry (packages/strategy/registry/src/index.ts). This is the only place an app names a concrete strategy (core invariant 1); apps/api and apps/worker consume the built registry, never a strategy-* package directly.
4. Add a golden-replay fixture and test. Capture a .jsonl fixture of real ticks and assert replayFixture(<strategy>, fixture) returns diff = 0, as momentum does with fixtures/replay/cross-cycle.jsonl. The replay gate diffs each tick's decisions and nextState against the fixture, a required quality gate that catches behavioural drift on every tick. It deliberately ignores logs and metrics — a strategy may add or reword either without re-capturing a fixture. That exemption is about the replay gate only; metrics still has to clear step 5.
5. Declare any new metric names. Every MetricEntry a tick returns is drained onto the catalogued strategy_metric_total series with its name as a label, so an undeclared name would export under a label nobody is watching. packages/strategy/registry/__tests__/strategy-metric-names.test.ts walks the metric( call sites across packages/strategy/*/src/** and asserts the exact set of names, so a new one fails the build until it is listed and a removed one fails just as loudly. Promote a free-form tag onto the series only through apps/worker/src/metrics/catalog.ts — anything not in labelNames is dropped by the sink, silently and by design.

File: docs/architecture/reliability.md Copy Chapter

Reliability

The worker is crash-only: idempotent jobs, a version-aware per-symbol symbol_states CAS, and idempotent clientOrderId mean a restart resumes cleanly and never double-places an order. The worker pipeline covers the tick lifecycle and commit path; this page collects narrower self-healing mechanisms.

See also — the worker pipeline owns the other self-healing mechanisms:

• Market-data liveness — REST gap-fill on a closed socket, force-reconnect on a silently stalled one.
• Held-quantity reconciliation — four paths converge the strategy's heldQuantity claim onto wallet truth.
• Account-event idle clock — a second liveness clock that reconnects and resyncs a stream that pongs but delivers no events.
• Delisted-symbol self-heal — a symbol Binance no longer lists reaps its own flat auto-discovered binding at the tick boundary instead of dead-lettering.
• Unpermitted-symbol self-heal — a symbol the account has no Binance permission to trade retires the same flat auto-discovered binding, before the tick assembler and at no request weight of its own.

Binance clock-drift self-heal Copy Chapter

Signed Binance calls carry a server-time offset added to every timestamp. When a call returns -1021 (timestamp outside recvWindow), the client issues one GET /api/v3/time resync, coalesced so a burst of concurrent callers triggers a single round-trip, then retries the call once. A second -1021 propagates to the caller so a persistent host-clock skew cannot loop (the operator fixes the clock). The retry is safe: Binance rejects at its timing gate before executing, so nothing was placed, and order placement is idempotent via clientOrderId. The executor treats any -1021 that surfaces past this as non-retryable, so the client owns the recovery and the two paths never stack.

flowchart TD
    Call["Signed Binance call"]:::io
    FirstGate{"-1021 timestamp error?"}:::gate
    OkA["Success"]:::act
    Resync["Resync GET /api/v3/time, coalesced"]:::core
    Retry["Re-sign and retry once"]:::core
    SecondGate{"-1021 again?"}:::gate
    OkB["Success"]:::act
    Propagate["Propagate to caller; executor treats it as non-retryable"]:::wait

    Call --> FirstGate
    FirstGate -->|no| OkA
    FirstGate -->|yes| Resync
    Resync --> Retry
    Retry --> SecondGate
    SecondGate -->|no| OkB
    SecondGate -->|yes| Propagate

    classDef io fill:#2c3e50,color:#ffffff;
    classDef core fill:#1f6f43,color:#ffffff;
    classDef gate fill:#f0ad4e,color:#2c3e50;
    classDef act fill:#1f6f43,color:#ffffff;
    classDef wait fill:#ecf0f1,color:#2c3e50;

Delisted-symbol self-heal Copy Chapter

Self-heal here means the tick fixes the condition itself and carries on, rather than failing the job into the dead-letter queue for a human.

A tick reads its symbol's filters from Binance exchangeInfo. When a symbol is missing, the tick first refreshes exchangeInfo inline (a brand-new worker may simply not have primed it yet). If that refresh resolves and the symbol is still absent, Binance no longer lists it on this account's mode (a delisting, or a symbol admitted to the wrong mode) — a SymbolDelistedError. This is distinct from a transient read failure (the refresh threw, a Redis error): those stay a bare Error and still dead-letter, because they may clear on their own.

For a held or operator-pinned delisted symbol — left in place (below) and so raising SymbolDelistedError on every tick — the inline exchangeInfo re-fetch is bounded to once per negative-TTL window (SYMBOL_INFO_NEGATIVE_TTL_MS, equal to the positive cache TTL): within the window the tick throws from memory without re-running the ungoverned full exchangeInfo fetch, keeping it off the hot path. The typed error still raises each tick, so the reap and action-log behaviour below is unchanged; after the window the next miss re-fetches, so a re-listed symbol still recovers.

On a SymbolDelistedError the tick self-heals instead of dead-lettering the same symbol every tick forever:

• It reaps the binding only when it is flat and auto-discovered (source=auto, no held quantity, no open order) via the shared reapAutoBinding — the same reap the discovery cron uses, so the two paths cannot leave divergent discovery state. The unbind itself deletes the symbol's condition_states, symbol_states, avg_entry_prices and pending override_actions in the same transaction, so a delisted coin leaves no state a tick could never come back to close. A held or operator-pinned symbol is left in place.
• It writes one operator action_log: info when the binding was removed; a warn when it was held or pinned (the operator must flatten or unpin it), throttled to one per hour per profile+symbol so a stuck symbol cannot flood the log.
• On a removal it enqueues one reconfigure-profile resync — the same job the discovery cron and the api symbol routes use — so the WS subscriber drops the now-unbound symbol promptly instead of feeding it until the next discovery pass. The payload carries accountId (a resync missing it fails the job as an invalid payload and dead-letters on that failure, rather than completing silently). The enqueue is a queue add, not a re-entrant tick, so it cannot deadlock the per-(profile, symbol) chain.
• It returns a graceful skip (no decisions, no dead-letter); the next market event re-ticks.

The action_log and the reconfigure enqueue are best-effort — a transient fault is logged and swallowed, never allowed to re-fail the tick the self-heal exists to rescue.

Unpermitted-symbol self-heal Copy Chapter

Binance gates each symbol on permission tags: a symbol publishes permissionSets, and the account may trade it only when it holds at least one tag from every published set. A bound symbol the account cannot satisfy is refused -2010 This symbol is not permitted for this account on every order it will ever send, and no amount of waiting changes that. So the tick retires the binding on the same terms as a delisting.

The check is data-driven, not error-driven — nothing throws — so it runs before the tick assembler rather than in its catch, and it spends no Binance request weight of its own: the symbol's sets come from the same symbol-info cache the assembler reads on the very next line, and the account's tags from the Redis key every signed /account response already writes. It is gated on the kill switch and the per-symbol pause, the two halts the assembler checks before its own symbol-info read, so running ahead of the assembler cannot retire a binding on a profile the operator has explicitly stopped.

It fails open in every ambiguity — the symbol publishes no sets, its published shape drifted, or the account's cached tags are absent, empty or unparseable — and the tick proceeds normally. The asymmetry is deliberate: a wrong "not permitted" silently retires a binding the account can trade, while a wrong "permitted" costs one Binance rejection, which is what happens today anyway.

When the symbol is confirmed unreachable, the tick takes the same self-heal the delisted path takes, sharing one implementation so the two cannot drift:

• Reap only when flat and auto-discovered, through the shared reapAutoBinding. The unbind deletes the symbol's condition_states, symbol_states, avg_entry_prices and pending override_actions in one Postgres transaction; the discovery Redis hashes are cleared straight after, and only on a confirmed removal. Held or operator-pinned bindings are left in place — retirement is never allowed to abandon a position or override the operator.
• One operator action_log: info on removal; a warn when it was held or pinned, naming the sell-down or unpin the operator has to perform. The warn is throttled to one per hour per profile+symbol on its own Redis key namespace — not the delist throttle's, and not the placement refusal's. All three are keyed (profile, symbol), so a shared prefix is a shared key, and the placement refusal always fires first.
• Only the sets the account satisfies nothing from are recorded, plus counts. Binance controls permissionSets and caps neither level, so a tokenised-equity symbol publishes hundreds of tags that no operator can act on and that the log viewer would then serve on every page.
• On a removal, one reconfigure-profile resync so the WS subscriber drops the now-unbound symbol, then a graceful skip: no decisions, no dead-letter.
• A binding that survives (held or pinned) keeps ticking. Only a retired symbol has nothing left to tick; one still bound must close its own blocker rows and stay able to cancel its resting orders.

Two boundaries are load-bearing. A SymbolDelistedError raised while reading the symbol row propagates to the delisted branch instead of being swallowed here, so a symbol Binance no longer lists at all is described accurately. And the order-placement pre-call refusal stays exactly as it is: retirement fails open in every ambiguity above and deliberately declines manual and held bindings, so that refusal is precisely where those still land, at zero request weight.

File: docs/architecture/tech-stack.md Copy Chapter

Tech stack & deployable components

What this repo is built from and what actually runs on a server. Versions are pinned in package.json and the lockfile — that is the source of truth; the ranges below are the declared ones at the time of writing and may have moved.

What runs on a server Copy Chapter

Three long-lived processes and two data stores. In the default deployment the three processes are one container.

flowchart TD
    Browser["Operator browserReact SPA, installable PWA"]:::ui
    App["app container, ROLE=allapi + live worker + study"]:::core
    Postgres["Postgres 17 + TimescaleDBdurable state, candles, ledgers"]:::io
    Redis["Redis 8queues, caches, market data"]:::io
    Binance["BinanceREST + WebSocket"]:::ext
    Notify["Slack / Telegram / webhook"]:::ext

    Browser --> App
    App --> Postgres
    App --> Redis
    App <--> Binance
    App --> Notify

    classDef ui fill:#ecf0f1,color:#2c3e50;
    classDef core fill:#1f6f43,color:#ffffff;
    classDef io fill:#2c3e50,color:#ffffff;
    classDef ext fill:#7C4A9E,color:#ffffff;

The web app is build-only — there is no web container. The api serves the built SPA same-origin, so the browser talks to one host.

The three roles Copy Section

Splitting them is deploy/compose/docker-compose.scale.yml, which parks app and starts api, worker, and study from the same image.

Ports Copy Section

The stack serves plain HTTP and does not bundle a reverse proxy — put TLS in front of it. See Production deploy.

The stack, by layer Copy Chapter

Runtime and tooling Copy Section

Backend Copy Section

Frontend Copy Section

The web app carries no decimal.js. Money arrives from the api as pre-formatted strings, so summation always happens server-side.

Observability Copy Section

Repository layout Copy Chapter

apps/
  api/       HTTP api + OpenAPI + SPA serving
  web/       React SPA (build-only, served by api)
  worker/    live trading loop, crons, backtest consumer
  server/    ROLE-selectable entrypoint + the one Dockerfile
packages/
  strategy/core/          the plugin contract + Executor
  strategy/trailing-trade/  strategy plugins, one package each
  strategy/momentum/
  strategy/rebalance/
  indicators/             pure Decimal-typed candle and window math
  notify/                 notifier contract, registry, providers/{slack,telegram,webhook}
  binance/                Binance REST + WebSocket client
  db/                     schema, repos, hand-authored SQL migrations
  contracts/              Zod schemas shared by api, web, and worker
  config/                 shared build/test config
  core/                   shared runtime utilities, exposed as subpaths
deploy/compose/           the compose files
scripts/ci/               invariant gates CI runs

Directory grouping under packages/strategy/* is filesystem-only — package names stay flat (@app/strategy-trailing-trade).

The two extension points Copy Chapter

Both are plugins behind a contract, so adding one is a new package plus a registry entry, never an edit to apps/api or apps/worker.

• Strategies — see Strategy plugin contract.
• Notifiers — see Notifiers.

Deliberate omissions Copy Chapter

• No encryption at rest. Binance keys and notifier secrets are stored as plain text. The mitigation is a single-tenant deployment plus an operator-side Binance IP allow-list. See Auth & threat model.
• No distributed locks. Single-execution comes from BullMQ job-id coalescing, in-process key ordering, idempotent client order ids, and version-checked state writes. See Reliability.
• No row-level security in the database. Account isolation is enforced by a branded two-tier scope in TypeScript plus a lint gate. See Account isolation.

File: docs/architecture/worker-pipeline.md Copy Chapter

Worker pipeline

Canonical references for the worker tick pipeline — the in-repo files are the source of truth, this page summarises the constraints those files encode.

• apps/worker/src/ — BullMQ-driven tick handlers; per-profile work runs serially via the in-process chainByKey Promise chain.
• packages/strategy/core/ — the Strategy contract and Executor that drives tick() and consumes the Decision union.
• packages/strategy/trailing-trade/ — the first packaged strategy, a trailing-trade strategy.

Single-replica policy Copy Chapter

The worker runs a single replica today on purpose: no Redis-backed distributed lock, no intent set, no soft balance reservation. Single-execution per profile is enforced by three composing primitives:

1. BullMQ jobId coalescing — the queue rejects duplicate tick jobs for the same profile while one is in-flight.
2. In-process chainByKey — successive ticks for the same profile serialise on a Promise chain so one replica never runs two ticks concurrently.
3. Idempotent clientOrderId — Binance dedupes order placements if the worker crashes between submit and Redis-side acknowledgement.

BullMQ is pinned to 5.58.6 exactly, not to a range. Every coalescing key above is colon-delimited (tick::<symbol>, reconcile-symbol::<sym>, archive-grid::<sym>:<ms>), and 5.58.7 added validation that rejects a custom job id containing : — BullMQ builds its own Redis keys around that separator, so an id carrying one corrupts the key it is spliced into. Redis itself treats a key as an opaque string. Upgrading past 5.58.6 therefore breaks primitive 1 outright, and with no distributed lock there is nothing to fall back on: the redesign of the key scheme has to land first. renovate.json holds the ceiling as an allowedVersions rule so the bot cannot propose the bump; a hand-edit is not gated, which is why it is written down here.

Multi-replica is not yet enabled; the elastic worker pool is built ahead of it: tick/cron jobs distribute via BullMQ competing consumers, and each account's user-data stream is owned by exactly one pod, elected by rendezvous (HRW) hashing over the Redis membership registry (ownership is a pure function of the live member set — no lock, no held key). The propagation seams land ahead of the flag but stay dormant at single replica: the fleet-global membership reconcile (above) and the strategy's optional mergeConcurrent latch-merge on a cross-pod tick-commit CAS miss (see the strategy contract). Reintroducing redlock / Redlock / intents: Redis sets / soft balance reservation fails CI (scripts/ci/no-locks.sh).

Cron health observability Copy Chapter

The single replica runs about 20 crons, most self-rescheduling (the next run is enqueued only on the current run's terminal state). A self-rescheduling cron that stalls or dies stops re-arming, and nothing surfaced that until a downstream screen went empty.

Every cron handler is wrapped by withCronStatus (apps/worker/src/crons/cron-status.ts) at registration, so each run records its terminal outcome — { lastRunAtMs, status: 'ok' | 'error', durationMs, error? } — into the worker:cron-status Redis hash (field = cron name). The wrapper is transparent: a status-write failure is swallowed, and a handler error is recorded then re-thrown unchanged, so BullMQ retry and the self-reschedule loop see the real result.

A self-rescheduling cron that runs LONGER than its period is a separate failure from one that stops re-arming, and it used to be invisible. registerSelfReschedulingCron computes the next delay as periodMs - elapsedMs floored at zero, so an overrun of any size collapses to a back-to-back re-arm that looks exactly like a healthy fast loop from outside: the same queue depth, the same completed jobs, the same worker:cron-status row. One 15-minute sweep ran for 8h07m on that path and emitted nothing. The re-arm still collapses (overlapping runs would be worse), but the run now warns with {cron, periodMs, elapsedMs} and increments cron_overrun_total{cron} as well. Both fire from reschedule, which only the completed and terminal-failed listeners reach, so this is a completion-time signal: it tells you a run WAS late, once that run ends. For the 8h07m case it would have arrived at 8h07m, not at minute 16. The in-flight indicator is the one already there, the worker:cron-status timestamp going stale on the System health panel, since a run still executing has not written its terminal outcome. The two are complementary and neither replaces the other. The diagnostics are emitted AFTER the re-arm and inside a try/catch: reschedule is the only place a self-rescheduling cron enqueues its successor, so an observability write that threw ahead of it would stop that cron until the next restart.

GET /worker/crons reads the hash and the Account page's "System health" panel renders each cron's last-run age + ok/error. A stale timestamp on a frequent cron is the "this cron has stalled" signal; an error row carries the failure message. This pairs with the worker heartbeat (worker:status, surfaced as "Bot live/down" in the account-health bar) and the DLQ job-failed alert: heartbeat answers "is the worker up", cron-health answers "is each job running", and the alert answers "did a job die".

In-process snapshots Copy Chapter

Two reads are shared process-wide rather than performed per consumer, both built once in buildBootContext and handed to every caller as a closure: getAssetPolicy (Binance's stablecoin/fiat classification, one HTTP fetch behind a five-minute snapshot) and getSymbolAdmission (the mode-keyed exchangeInfo admission map, one paged SCAN + MGET walk over the whole binance:symbol-info* keyspace behind a two-minute snapshot). The consumers are the discovery-run cron and the diagnosis re-probe, which re-derives the same candidate funnel live so the operator can see exactly where a symbol was cut.

Sharing buys two things. The first is agreement: two independently built maps can classify one asset, or admit one symbol, two ways at the same instant, and a probe that disagrees with the cron describes a funnel the cron never ran. The second is cost. The admission sweep pages roughly 1.4k keys, and the probe was firing its own sweep moments after the cron had finished one.

The probe therefore moved from always-fresh to bounded-stale, and that is the trade being made. ADMISSION_SNAPSHOT_MAX_AGE_MS is 120s: above the 60s discovery wake, though that lower bound buys less than it looks like — each profile runs on its own refresh period (15 minutes by default) and a wake with no profile due never builds the map, so most wakes prime nothing and a probe arriving at an arbitrary moment still sweeps for itself. What reliably holds is a probe landing within 120s of a cycle that actually built the map, which reuses that build, and repeated probes, which collapse onto one sweep. The upper bound is the firm one: below the 5-minute exchange-info-refresh cadence that writes those keys, so one snapshot spans at most one write and a delisting cannot stay invisible across two consecutive refreshes. The window must equal neither bound; a unit test pins it between them.

An empty map is not a snapshot. fetchSymbolAdmission returns one both for a Redis fault and for a keyspace that was never primed, so it is a sentinel meaning "unreadable or unprimed", never "this mode lists no symbols" — and the discovery handler treats it as a reason to skip the profile. Retaining it for the full 120s would keep aborting cycles long after a one-second blip healed, so an empty result is memoized for EMPTY_MEMO_MS (30s) instead. What that window guarantees is its own length, not that a wake boundary clears it: the memo is stamped when the sweep resolves, the cron self-reschedules at max(0, period - runtime), and the probe stamps it from outside any wake, so the honest statement is that a sentinel is served for at most 30s and at most one subsequent wake can reuse it. The window's beneficiary is the on-demand probe rather than the cron, which already coalesces to one admission read per mode per wake in its handler; the cost is that a probe inside the window answers from the stored sweep even if Redis has recovered. A later successful sweep replaces the sentinel outright.

An empty map is also no longer merely "skip this profile this wake": the discovery handler counts an empty-admission-map abort and parks it where the operator's diagnosis page reads it. A memo can therefore extend a real refusal by at most one wake — the sentinel is only ever served after a sweep actually returned one, so nothing is invented from a healthy Redis — and the parked abort is cleared on the next successful cycle, by which time the memo has expired anyway.

Snapshots are per mode and never substituted for one another: a live map served to a testnet profile binds symbols testnet does not list, and every tick for them then DLQs. The two modes are disjoint by key glob (binance:symbol-info:* vs binance:symbol-info-test:*), so one sweep cannot see the other's universe. Concurrent readers collapse onto one in-flight sweep per mode, which is the case the sharing exists for: the cron's profile loop is sequential, but the diagnosis queue worker runs concurrently in the same process.

The wake-scoped admissionByMode memo inside the discovery handler stays as it is. It guarantees that every profile in one wake sees the identical map, which a time-bounded snapshot does not: a wake that straddles the TTL would otherwise hand its later profiles a newer map than its earlier ones.

One log-attribution detail changed with the sharing. fetchSymbolAdmission takes a caller tag for its two warn lines (unreadable keyspace, empty map), which used to read cron discovery or diagnosis. A memoized sweep has no single caller — the warn belongs to whichever consumer happened to miss the snapshot — so the tag is now the construction-time symbol-admission for both. Read it as a process-level fact about the keyspace rather than as an attribution: the line names no consumer, the logger carries no run or cron id, and the cron and the probe run concurrently here, so a neighbouring line does not identify one either.

Market-data subscriptions Copy Chapter

For each trading symbol the worker opens the union of its profiles' feed intervals — each profile's trading interval plus 1m (bounds currentPrice staleness to ≤1 minute so a stop-loss fires within the minute, not a full trading-candle late) and 1d (daily regime / all-time-high refresh) — plus one mini-ticker. Each WS kline stream is refcounted by the number of profile claims (apps/worker/src/market-data/subscriptions-manager.ts): a stream opens on the first claim and closes only when its last claimant leaves, and the mini-ticker lives as long as any kline interval does. So two profiles trading the same symbol on different intervals each get their own interval subscribed, while a shared interval (1m/1d, or two profiles on the same trading interval) is opened once and refcounted.

The subscription set equals the per-tick candle-load set: both come from one feedIntervals helper (apps/worker/src/market-data/feed-intervals.ts), so the streams a symbol subscribes and the candle windows a tick reads never drift. An unsubscribed interval would otherwise trigger a per-tick cold-load REST fallback.

ProfileManager (apps/worker/src/profile-manager/profile-manager.ts) forwards each profile's full symbol delta with its interval to the subscriptions-manager and keeps the per-symbol inverse index only for the profilesUsing event fan-out; the WS-stream refcounting lives entirely in the subscriptions-manager.

WS connection pool (sharding) Copy Section

A single Binance combined-stream socket caps at 1024 streams. The kline-fetcher (packages/binance/src/market-data/kline-fetcher.ts) therefore holds a pool of connection members, not one socket. Each member owns its own socket, RPC buffers, reconnect-backoff cursor, lastFrameMs, and the set of streams it carries.

• Assignment is greedy. A new stream goes to the first member with spare capacity (streams.size < 1024), else a new member is allocated. The 1024 cap is enforced at assignment time, before any SUBSCRIBE is sent, so a member's stream set can never exceed it; connect's ensureLimit is unreachable defense-in-depth.
• Hard ceiling. The pool tops out at BINANCE_MAX_POOL_MEMBERS = 16 (16 × 1024 = 16384 streams ≈ 4096 symbols at 4 streams/symbol). The next stream past the ceiling throws loudly rather than silently dropping a subscription — a symbol set this large needs the deferred consistent-hash sharding, not a wider single-process fan-out. 16 combined-stream connections sits well within Binance spot's per-IP connection limit (300 new connections per 5 minutes per IP per the spot WebSocket docs), so the ceiling bounds stream count without risking a connection-limit breach.
• Per-member lifecycle. Reconnect (exponential backoff, reset only on a real delivered frame), resubscribe-on-reconnect (rebuilt from that member's own streams), and the onReconnect fire-once-per-member are all scoped to one member. A close or stall on one member reconnects and resubscribes only its own streams — a sibling member is never touched, and adding a shard never fires a spurious whole-feed resync.
• Member pruning. When a member's last stream unsubscribes, its socket closes and the member is dropped from the pool. Members are addressed by stable id, not array index, so pruning never reroutes another member's keys.

The pool is invisible to callers: the KlineFetcher interface is unchanged, and its aggregate public methods collapse the pool to the single observable contract the subscriptions-manager and liveness watchdog already consume:

• subscriberCount / activeKeyCount — sum across members (same totals as the single-socket era).
• isConnected() — all-open: true iff every non-empty member is OPEN. A partially-stalled pool (one member down while others serve) reads as not-whole so the watchdog recovers the whole feed rather than treating it as healthy. Empty pool → false.
• msSinceLastFrame() — worst case: max over members of now - member.lastFrameMs, so a single stalled member is enough to trip the watchdog's stale threshold.
• forceReconnect() — force-reconnects every open member, recovering the whole feed in one watchdog pass; a member that is mid-connect or already closed is left to its own onClose-armed reconnect.

For the overwhelmingly common case (≤256 symbols → one member) the pool is a single member created first, so behaviour is byte-for-byte identical to the prior single-socket adapter.

Market-data liveness Copy Section

Tick cadence is 100% WS-frame-driven, so if the market-data feed stops, every in-process stop / trailing / time-stop / entry freezes for that symbol. Two layers keep the feed alive (apps/worker/src/market-data/market-liveness-watchdog.ts):

• Disconnect (socket closed). While klineFetcher.isConnected() is false, the watchdog REST-polls each subscribed symbol's freshest closed-1m price and feeds it back as a synthetic mini-ticker through the same onMarketEvent path a real frame uses, so the strategy keeps evaluating a fresh price during the reconnect-backoff window. Idempotent (same coalesced tick job, deterministic clientOrderIds) and inert while healthy.
• Silent stall (socket open, no frames). A half-dead socket leaves isConnected() true, so the disconnect layer alone would never fire. The kline-fetcher tracks msSinceLastFrame() (bumped on every frame; the always-on mini-ticker pushes about one frame per second, so it is the natural heartbeat); when it exceeds staleThresholdMs (default 20s, floored 10s) with active subscriptions, the watchdog calls forceReconnect(), which closes every open pool member so the normal per-member reconnect+resubscribe path rebuilds the whole feed. msSinceLastFrame() is the worst case across members, so a single stalled member trips the threshold; the close flips the all-open isConnected() false synchronously, so the same watchdog pass falls through to the REST gap-fill. The reconnect backoff resets only on a real delivered frame (not merely on open), so a persistently open-but-silent upstream backs off toward the cap instead of looping at the initial delay. This mirrors the user-data stream's heartbeat + idle-watchdog.

Subscription lifecycle Copy Chapter

subscribe-profile, unsubscribe-profile, reconfigure-profile, and dispose-profile all dispatch under chain.run(profileId) (apps/worker/src/queues/pipeline-worker.ts), so under pipeline concurrency they cannot interleave on the same profile. Job arrival order does not decide the outcome; the operator's last committed DB action does.

`dispose-profile` — deleting a profile without abandoning its exposure Copy Section

The api guards and enqueues (202); this job owns the teardown, because only the worker has a Binance client and a profile with resting orders cannot simply be dropped from the DB — the orders survive on the exchange, holding the operator's coins, with nothing left pointing at them. Crash-only: every step re-derives from DB truth, so a kill mid-teardown resumes on the retry. The order is load-bearing:

1. setEnabled(false) + unsubscribe first — a live tick would otherwise re-place what we are about to cancel.

2. Cancel each live order on Binance (there is no cancel-all endpoint, so it is a loop). An already-gone order is tolerated.

3. handoff only: re-point the position (avg_entry_prices + the profile_symbols binding) to the target profile, never the orders, then reconfigure the target so it ticks the symbol and seeds its strategy state from the moved cost basis. Two properties make this safe, and both were bugs first:

   • It is ONE transaction. The source must be unbound before the target is bound (profileSymbols.upsert enforces base-asset exclusivity per account), and that unbind destroys the row the retry rebuilds its plan from — so a crash in between would leave the retry with nothing to hand off, the profile deleted, and the cost basis cascaded away: coins in the wallet owned by nobody. withTx re-brands both already-proven scopes onto one transaction handle, so a crash rolls back to a fully intact source or commits a fully seeded target.
   • The plan is the UNION of the symbol bindings and the cost-basis ledger. Either half can exist without the other, and both mean "this profile owns these coins": a ledger row with no binding (a symbol un-bound while a position was open), and a binding with no ledger row (a HELD BUT UNPRICED position, e.g. cost-basis reconstruction that never succeeded because getMyTrades kept failing). Dropping either from the plan hands those coins to nobody and lets the delete cascade the last row pointing at them away. The binding's reserve_base_quantity travels with it, or the target would treat the operator's ringfenced coins as tradeable position.
   • The seeding is VERIFIED, not assumed. reconfigure's reconcile is fail-soft by design (a missing symbolInfo key, a getAccount blip, or a strategy without a position adapter must never crash a live profile's reconfigure), so it can seed nothing and still return. Every other step of the disposal is held to "throw until provably clean", and this is the one whose failure is UNRECOVERABLE — delete the source and the target holds the coins while reading FLAT, with no row left for a retry to re-derive. So before the delete, every cost-basis row the TARGET holds must read back as a priced position through that strategy's own position.readPosition adapter; anything else throws and BullMQ retries (the retry re-runs reconfigure and re-verifies — the check is driven by the target's ledger, never by this run's plan, which is empty by then).
   • State seeding does not depend on ProfileManager membership. The natural handoff target is a fresh or stopped profile, which is absent from ProfileManager's in-memory map. reconfigure therefore runs the wallet reconcile BEFORE its membership gate, and ensureCostBasisFromTrades seeds the state body straight from the cost-basis ledger when the target has never ticked the symbol (the reviver restores a price onto a body and has none to work with). Otherwise the target reads FLAT while holding the coins: no protective stop, and a fresh entry BUY on top of the position it already owns.
   • A shared-wallet collision on the target is a CLEAN BLOCK, not a retry. The bind is profileSymbols.upsert, which throws SymbolOwnershipConflictError (the target already trades that base asset) or SiblingQuoteConflictError (a target sibling settles in it) — the same account-isolation guards docs/architecture/account-isolation.md describes. These are TERMINAL: no retry clears a wallet line another profile owns, so rethrowing would dead-letter the job and retry the whole 10-minute envelope to no effect while the source profile could never be deleted. The handler catches BOTH classes around the handoff transaction (which has already rolled back, so the source keeps every position and its row) and acks. Because the finders exclude the target, the profile that actually owns the asset is a third sibling carried on the error, so the notice names that owner — not the target — and states plainly that the source was already disabled and its orders cancelled, leaving its position unprotected: the operator re-starts it to restore protection, then picks another target or frees the asset and retries. Only these two classes ack; every other throw from the handoff still propagates and retries.

4. Re-read DB and Binance and confirm the exchange is clear. The Binance read is one account-wide getOpenOrders() — no symbol argument (weight 80, against 6 per symbol). Asking per symbol could only ever ask about symbols the DB still knows, and the order that most needs finding is precisely the one our books never recorded: the bookkeeping-failure path leaves a live order with no row, and a later unbind takes away the last symbol that would have made a per-symbol probe ask. The whole book is then split three ways:

   • Still resting (an order we just cancelled, matched by symbol + orderId — a Binance order id is unique per symbol, not per account): the cancel never landed. Throw, and BullMQ retries; the profile outlives it.
   • Untracked but provably ours: cancel it. Ownership rests on two independent proofs, and either suffices. (a) attributeOrder (the same proof the adopt page uses, run against the symbol's merged config, since a per-symbol override widens the id space the strategy enumerates) re-derives the clientOrderId. (b) A row this profile recorded, matched on symbol + binanceOrderId with no closed_at filter, is proof of ownership a re-derivable id cannot give: upsertLive's closePrevious stamps the prior slot row CLOSED the moment the next candle's order takes it, while the superseded order may still rest, and a strategy whose id folds unbounded runtime data (momentum's candleCloseMs) cannot re-derive it at all. Leaving either is exactly how a deleted profile's stop went on holding the operator's coins for days with nothing left in the system pointing at it.
   • Unattributable (an operator's hand-placed order, a sibling profile's, or one whose id folds unbounded runtime data and has no recorded DB row): announced to the operator, never cancelled, and it never blocks the delete. We do not cancel an order we cannot prove is ours. The announcement is scoped to the symbols this profile kept — a sibling's resting order is not this profile's leftover, and neither is the stop the handoff target armed seconds ago — and it names whether the order holds coins (a SELL) or cash (a BUY). It runs before the cancels, so a transient cancel failure cannot swallow the alert.

   An unresolvable Binance client is a failure, not a pass — "I could not ask" is not "it is clear".

5. Only then wipe Redis and delete the row — in that order, so a crash between them leaves a live profile that the retry re-derives, not a live profile with no cached state.

Every failure above throws — except a shared-wallet collision on a handoff target (step 3), which is terminal and acks cleanly after notifying the operator — so BullMQ retries (DISPOSE_JOB_OPTS, defined in apps/api/src/route-helpers.ts because the api enqueues this job even though the worker owns the teardown: 8 attempts, exponential backoff from 5s ≈ a 10-minute envelope, sized against a real Binance outage rather than a blip — and a BullMQ job runs exactly once unless attempts says otherwise) and DLQs only after the retries are spent. It carries no fixed jobId (the pipeline queue retains completed jobs, so a fixed id would let only the first disposal per profile ever run). The per-profile chain key does not serialise it against the tick (which chains on ${profileId}:${symbol}); step 4 is what closes that race — a tick that slipped an order in after the cancels is found by the DB+exchange re-read, throws, and the retry cancels it.

handleUnsubscribe re-reads the DB and tears down only when the profile is missing (deleted) or enabled=false; otherwise it skips. A /stop's unsubscribe can land after a later /start has already re-enabled the profile, and tearing down then would strand the symbol (DB enabled, streams gone) until restart. The ownership read throws ProfileNotOwnedError when the row is gone, which the handler maps to the deleted-profile teardown path rather than DLQ-ing.

enable() on an already-active profile converges instead of no-opping: it applies the current symbol set and interval via setSymbols and refreshes the technicals intervals. It does not touch the account user-data stream — ProfileManager owns only market subscriptions and the membership set (listActive). Opening and closing the user-data stream is subscription-ownership's sole responsibility, elected by HRW over listActive().

Sole stream driver + fleet-global membership Copy Section

subscription-ownership is the only thing that opens or closes an account's user-data stream. At single replica the sole ready member owns every account, so its first reconcile opens all streams at boot; on unsubscribe it closes the departed profile's stream (tracked in a streamedByUs map, since a profile that left listActive is no longer visited by the open/close election loop).

Because HRW ownership elects over listActive(), every pod's membership must reflect the fleet-global enabled set. subscribe/unsubscribe/reconfigure arrive as single-consumer pipeline jobs, so after boot only the consuming pod knows a change. The EnabledSetReconciler (apps/worker/src/profile-manager/enabled-set-reconciler.ts) closes that gap: on an interval it re-reads repo.profiles.listAllEnabled and converges each pod's ProfileManager via the diff-based reconcile(rows) (enable added, disable removed, converge the rest), then re-elects ownership. The subscribe/unsubscribe handlers additionally kick reconcileOwnership() for prompt convergence (best-effort; the periodic reconcile is the guarantee). The reconciler and ownership both run at single replica (started for the worker role — ROLE=worker or ROLE=all), where the pipeline job has already applied the change on the sole pod, so each pass is no-op churn; the election only redistributes streams across pods once replicas > 1 is enabled. This is the multi-replica propagation prerequisite.

Converging on exchange truth Copy Chapter

The strategy's heldQuantity is a claim; the wallet is the fact. When they disagree — a fill the user stream never delivered, an order the operator cancelled on Binance by hand — every SELL the strategy sizes off the stale claim is refused -2010 (insufficient balance), forever, because nothing in the tick path ever revisits the claim. Four paths converge it, in ascending order of latency and descending order of precision.

All four run the same runHeldQuantityReconciliation / reaper code under the same profileId:symbol chain key, so none of them can interleave a state write with a live tick on that symbol.

The periodic backstops Copy Section

Both crons use selfReschedulePeriodMs, not a cron pattern: each fans out a getAccount per profile plus trade-history reads per symbol, so a slow run must delay the next one, never overlap it. Two reconciles racing the same symbol would contend on the shared chain key and burn Binance request weight re-deriving the same answer.

They are also kept separate rather than folded into one "converge to exchange truth" cron: they converge different things against different Binance endpoints, and a reaper fault (a getOrder 5xx storm) must not delay the position convergence, which is the money-critical half.

Neither can be boot-only. A healthy worker does not restart, so "reconcile at boot" means "reconcile never" — which is exactly how a stale heldQuantity or an orders row left at status='NEW' survives indefinitely. A stale order row is not merely cosmetic: it is shown as open in the UI, counted toward the account's open exposure (which gates account deletion), and treated as tracked-live by the orphan detector, which is what keeps the real order from ever being surfaced.

The `symbol-reconcile` queue Copy Section

The fast path. Three causes enqueue it, each a moment where the bot has just PROVEN its own state wrong:

1. a cancel whose -2011 probe comes back FILLED — the order filled during a stream gap;
2. a SELL Binance refuses -2010 — the wallet does not hold what the state claims. -2010 is NEW_ORDER_REJECTED, an umbrella code, so this is narrowed by the rejection's message: the symbol is not permitted for this account flavour enqueues nothing. That refusal says nothing about the wallet, so reconciling would find no drift to fix, and it never clears on its own, so every tick would re-enqueue the same futile pass forever. The tradability pre-flight and its once-per-hour-per-symbol alert own the moment; the binding itself is retired at the next tick boundary, or left in place with its own hourly warn when it is held or operator-pinned — see Unpermitted-symbol self-heal;
3. the account-event idle watchdog firing (below), which fans out one enqueue per symbol on the profile with cause stream-silent.

It is a queue, not an inline adopt, because the discovering code runs inside the tick's chainByKey(profileId:symbol) critical section and the fill-adopter takes that same key. chainByKey is not reentrant, so adopting inline would self-await and hang that symbol's tick forever. The work is deferred out of the lock and done here.

Enqueues are coalesced two ways. BullMQ's jobId (reconcile-symbol::<sym>) collapses a burst into one run while a job is waiting or active — paired with removeOnComplete: true / removeOnFail: true, without which the retained terminal job would keep occupying the id and the slot would never reopen. Beyond that, a 60s SET NX PX window per (profile, symbol, cause) bounds the treadmill: the discovering conditions fire EVERY TICK for as long as they hold, and some hold for days, so an unthrottled enqueue would spend getAccount + getMyTrades weight once a second on a symbol that is stuck. Dropping a reconcile inside the window is safe — the pass is an idempotent converge-to-truth, and the 15-minute cron is the backstop it relies on. The cron path is deliberately not throttled.

The account-event idle clock Copy Section

The user-stream pool runs a second liveness clock, independent of the heartbeat one. Binance's pongs prove the socket is up; they prove nothing about whether events are still being delivered on it. A silently half-dead stream answers every ping and delivers no executionReport, and the bot happily trades on a position it no longer holds.

So the pool also tracks time since the last account event per profile. Past accountEventIdleMs (default 45 min, floor 60s) it reconnects, resyncs, and raises onStreamSilent — which writes a durable operator-visible action_logs row (throttled per (profile, topic)) and enqueues a symbol-reconcile for every symbol on the profile.

Silence here is not a fault: Binance emits an account event only when a balance changes, so a profile holding through a quiet market is legitimately idle for hours. The operator-facing copy says what actually happened — quiet stream, reconnected to check — rather than "the stream is broken", which would train the operator to ignore the one row that, on the day it matters, is the only warning a fill went missing.

Boot & reconfigure Copy Section

On boot (runHeldQuantityReconciliation, apps/worker/src/boot/reconcile-held-quantity.ts) and on a mid-run reconfigure after an operator adopts an orphan (reconcileSymbolsAfterReconfigure in apps/worker/src/queues/pipeline-worker.ts), the worker reconciles each (profile, symbol) under the profileId:symbol chain key so a boot-window fill cannot interleave. ensureCostBasisFromTrades runs first: for a held-but-unpriced position — the wallet holds the coin but both the strategy state and the avg_entry_prices ledger are empty — it reconstructs the average entry price from Binance myTrades (limit: 1000, average-cost method), upserts the ledger, and applies a synthetic buy onto strategy state so the entry gate does not re-buy a position it already holds. Then reconcileSymbol pins heldQuantity to wallet truth and revives avgEntryPrice from the ledger. On a reconfigure this runs before setSymbols, so the newly-subscribed symbol is not yet tickable and there is no tick-vs-reconcile race. Both steps are best-effort: a getMyTrades / getAccount failure logs at warn and continues.

What counts as a position, and what is only residue

LOT_SIZE.stepSize is the smallest tradeable increment; NOTIONAL.minNotional is the smallest tradeable value. A balance can clear the first and fail the second by orders of magnitude: a crumb one step wide can be worth a fraction of a cent, and no sell will ever dispose of it. Seeding a position asks both halves; the flatten that removes one asks value alone, because an increment test cannot see a position whose wallet has decayed to residue.

Two bars, split by whether the decision creates a position or destroys one:

Declining to create costs only a blind spot on a balance nothing could trade anyway. Destroying a cost basis, or abandoning a position a ledger row already records, destroys something real, so the destructive paths take the stricter bar. Seeding from an existing row counts as destructive: that branch runs only when no state body exists, and a decline deletes the row.

Three rules bind any change here:

• The flatten tests the claim, not just the wallet. getAccount is read once per profile outside the per-symbol loop, while heldQuantity is read inside the per-symbol lock, so a BUY filling mid-sweep presents as a dust wallet under a fresh claim and must keep its cost basis. An unparseable claim counts as valueless; a claim of exactly zero is not a claim at all and falls through to the no-op band, since every idle symbol carries one.
• The flatten sits ahead of the |held − wallet| ≤ stepSize no-op band. That band is where a stranded position ends up: pinning heldQuantity to the wallet makes the two agree exactly, so every later pass short-circuits on a difference of zero before reaching any dust test.
• A missing input disarms the bound rather than guessing. These bounds only ever remove a position, so absence must mean "do not act", whether the symbol is unpriceable, minNotional is missing, or unreservedWalletTotal is.

Value bounds read unreservedWalletTotal, the wallet before the operator's reserve is drained out of it, for the reason given under Per-symbol reserve. Increment bounds read the adjusted balance instead, because no order can be placed for less than one step of the slice the strategy may trade. Where a rule combines the two, they are asked of different balances rather than both of the adjusted one.

A flatten clears strategy state first (applyFill({kind: 'empty'})), then deletes the avg_entry_prices row. The writes are not atomic and the order is deliberate: a lost delete is repaired by the next prune, but a lost clear leaves a claim nothing can heal. No archive-grid-trade is enqueued, matching the phantom prune: a flatten asserts the position never existed, so there is no exit fill and no realised P/L.

Prices resolve cache-first (readTickerPrice), then one batched getPriceTickers call per profile for whatever the cache missed. The REST half is not redundant: the miniTicker cache is written only by the market stream and its keys expire in 60s, so at cold boot, exactly when this sweep runs, it is empty for every symbol.

Both outcomes are observable. reconcile_value_bound_disarmed_total separates "checked, the holding is real" from "could not check", which previously both tallied as no-op; reconcile_position_removed_total records the two destructive paths.

The operator sets the average entry price via the API, which enqueues a pipeline job to update the strategy's position state. See the "Tell the bot the entry price" section in the troubleshooting guide.

Auto-archiving a closed cycle Copy Chapter

When a SELL fill empties a held position, the buy/sell cycle is over and its realised P/L should land in trade_archive without the operator clicking anything. Two files cooperate:

• apps/worker/src/executor/fill-adopter.ts — after the emptying SELL commits, it enqueues an archive-grid-trade pipeline job (enqueueArchive). This is best-effort: a queue failure is logged at warn and swallowed, because the fill is already durable and must not be undone by an archive hiccup. The jobId carries a per-fill timestamp so two separate exits on the same symbol each archive rather than coalescing into one.
• apps/worker/src/queues/pipeline-handlers/archive-grid-trade.ts — the handler. Because the same job can be delivered twice (BullMQ retry, or a manual operator click racing the auto-enqueue), it reads latestArchivedAt for the symbol and only archives orders that closed after that cutoff. A duplicate delivery finds nothing new and short-circuits with a nothing_to_archive log, so it is idempotent.

Netting the BUY commission out of the tracked quantity Copy Section

The archive only fires when the SELL resolves the position to clear, so anything that leaves a permanent residual silently suppresses the whole cycle's history. A base-asset commission does exactly that.

Binance charges a spot BUY's commission in the base asset (unless a discount asset such as BNB covers it), and the executionReport's z (executedQty) is the gross filled quantity, before that deduction. Folding z therefore tracks more coin than the wallet ever received. The exit is sized from the real wallet balance, so it can never bring the tracked number to zero: the SELL resolves sell-reduce, enqueueArchive never fires, and a completed, profitable trade never reaches Trade History. Live case: a BUY of 1682.30 TST paid 1.6823 TST in commission and credited 1680.6177; the protective stop sold 1680.60 (the wallet balance floored to the LOT_SIZE step), leaving 1.70 tracked — far above the 0.1 step, so the sub-step flatten never fired.

The netting happens in the adapter, before resolveFill is called, so the shared fold (and with it the backtest and the golden fixtures) is untouched:

• packages/binance/src/user-stream-frame.ts parses n / N (commission and its asset). Both are per trade while z / Z are cumulative, so a consumer needing the order's total must accumulate the partials itself.
• apps/worker/src/executor/order-commission-accumulator.ts does that accumulation per commission asset, keyed by (accountId, symbol, orderId) — Binance order ids are unique per symbol, not per account. Replay protection records every trade id and its fee rather than using a high-water mark: the user-stream pool dispatches handlers without awaiting, so a later trade can be folded before an earlier one. An identical reconnect replay is a no-op; a conflicting replay or malformed fee latches the order as unknown so a partial subtotal is never presented as complete.
• apps/worker/src/executor/fill-adopter.ts resolves the symbol's base asset and folds executedQty − commissions[baseAsset]. Other subtotals, such as BNB, stay in the record but do not reduce the credited base quantity. No base subtotal, an unresolvable base asset, an invalid record, or a net quantity ≤ 0 falls back to the gross quantity: absent means unknown, never a silent zero-fee assumption. The position's average entry price divides cummulativeQuoteQty by the net quantity, because that is what the operator paid for what they actually received.
• apps/worker/src/executor/fill-backfiller.ts builds the same per-asset totals from myTrades, so reconnect recovery and the live stream use the same fee contract and fail-closed behavior. Before aggregating, it validates the requested symbol, numeric trade/order identities, side, and non-negative decimal quantities, then deduplicates identical trade-id replays. An invalid row or conflicting payload for one trade id rejects the whole response before any order is adopted.

With the BUY folding 1680.6177, the live case's residual becomes 0.0177 — below the 0.1 step — and the existing sub-step flatten resolves it to clear on its own. When the LOT_SIZE lookup itself fails the residual cannot be checked at all; rather than stranding it silently, the adopter logs an error and appends a plain-language action log so the operator sees the unverified position.

Two filters make a residual unsellable, not one Copy Section

The same split applies to a sell's leftover: netting shrinks the residual but does not guarantee it lands under one step, and a crumb above the step can still be worth far less than one minimum order. resolveFill therefore takes minNotional alongside stepSize and flattens on either, valuing the residual at the exiting sell's own VWAP (cumQuoteQty / cumQty), the best evidence of what it is worth at the moment it would strand. A non-positive price skips the check, so a caller that does not price its sells cannot empty a live position through a zero. Both parameters are optional and the backtest passes neither, so the shared fold and the golden fixtures replay byte-identical.

The value bound additionally requires the leftover to be under 1% of the pre-sell position, and that qualifier is load-bearing: rebalance trims a holding to a target weight on purpose, and a small target weight is legitimately worth less than the floor, so flattening on value alone would delete a real cost basis and archive a phantom cycle. The increment bound needs no such qualifier, because an increment is price-invariant.

Flattening the fill is only half of it: reconcileHeldQuantity would read the same untradeable balance off the wallet on its next pass and re-create the position the fold just cleared, so both apply the same predicates. See What counts as a position, and what is only residue.

Cost-basis accounting Copy Section

Realised P/L is cost-basis-matched at fill time, not a window cashflow difference. On every SELL fill the fill-adopter computes realizedPnl = matchedProceeds − costBasisQuote against the position's avg entry price (realizedPnlOnSell in @app/strategy-core, the accounting counterpart to the resolveFill fold) and stamps realized_pnl / cost_basis_quote onto the orders row before enqueuing the archive. The archive aggregator (summarizeArchiveSince) then sums those columns: profit = Σ realized_pnl, total_buy_quote = Σ cost_basis_quote.

This is the fix for the phantom-profit class. The earlier aggregator differenced Σ sell − Σ buy over the orders closed since the last archive, which has no concept of cost basis: selling an adopted position (held base the bot never bought through an order row) booked the full proceeds as profit, and a hold spanning an archive boundary split wrong on both sides. Two properties make it fail-safe now:

• Overshoot is capped. Matched quantity is clamped to the held quantity, so selling more base than the bot tracks realises only the tracked portion (proceeds taken pro-rata); the un-costed surplus contributes nothing.
• Unknown cost basis never fabricates. A SELL with no ledger cost basis carries a NULL realized_pnl, excluded from the sum — a conservative under-count, surfaced via a pipeline_archive_grid_trade_missing_cost_basis warn, never a proceeds-minus-zero gain. The count is also persisted onto the row (trade_archive.missing_cost_basis, migration 0080) and carried on the API projection, because an under-count of zero is indistinguishable from a genuine break-even once written: without the flag the archive page renders a real trade as a confident +0.00. A row with a positive count shows P/L unavailable and an em-dash percent instead. The money columns are not a safe fallback either: total_buy_quote is Σ cost_basis_quote so an un-costed SELL adds nothing, and total_sell_quote is derived as total_buy_quote + profit, so a fully un-costed cycle renders 0/0 and the period rollups count it as zero. The inline explanation says exactly that instead of claiming the amounts are correct. A MARKET sell (the dominant exit) is inserted already-FILLED by place-order, so its stamp comes from stampRealizedPnl (status-independent), not the markFilled status flip; a boot-reclaimed sell the user-stream missed carries no stamp and is recovered by the backfill.

The per-intent:side breakdown jsonb is intentionally NOT cost-basis-matched — it remains a raw filled-quote (cummulativeQuoteQty) sum for the UI decomposition, so the breakdown totals are not expected to reconcile against profit.

Fee fetch Copy Section

The handler also records the Binance commissions paid during the cycle. It calls getMyTrades for the symbol (max page, limit: 1000), keeps only the trades whose orderId is in the archived set, and sums the commission per asset into the row's fees column.

Fees are a nicety, not a reason to fail an archive, so the fetch degrades rather than throws:

• If the per-profile Binance client is unavailable, the call fails, or a commission value cannot be parsed, fees is set to {} and a pipeline_archive_grid_trade_fees_unavailable warn fires. The archive still inserts.
• If the trade page was truncated and some archived orders matched no returned trade, the summed fees are a partial under-count. Those fees still insert, and a distinct pipeline_archive_grid_trade_fees_partial warn (with the count of unmatched orders) flags the gap so the under-count is never silent.

Fee reconciliation (backfilling `fees_quote`) Copy Section

The degrade-to-zero above, plus rows archived before the fees_quote column shipped, leaves some archive rows with fees_quote = 0 even though commission was paid — so net-of-fee P/L reads too rosy. An operator-triggered reconcile re-derives the real commission:

• POST /profiles/{profileId}/reconcile-fees (apps/api/src/routes/profiles.ts) — 202, enqueues a profile-scoped reconcile-fees pipeline job (jobId-coalesced).
• apps/worker/src/queues/pipeline-handlers/reconcile-fees.ts — lists trade_archive rows with fees_quote = 0, and for each matches the symbol's recent getMyTrades (limit: 1000) by the Binance order ids stored in the row's orders jsonb, exactly as the archive path does. Commission is valued in quote via the shared valueCommissionInQuote (re-used from archive-grid-trade.ts, so the two paths cannot drift), then written back through the profile-scoped p.tradeArchive.updateFees. Net-of-fee P/L (which subtracts fees_quote) corrects itself on the next dashboard/History read.
• Same limit: 1000 recent-page bound as the archive path: a round-trip older than its symbol's last 1000 trades matches nothing and is left untouched (counted unreconciled in the pipeline_reconcile_fees_done log), never overwritten with a fabricated zero. The Reconcile fees button lives in the profile Manage card, opened from the Manage profile slide-over on the overview.

Backfilling historic round-trips Copy Chapter

The auto-archive above only records cycles going forward and reads the local orders table. Round-trips that completed before it existed — or whose BUY rows are ghost-missing — leave trade_archive empty, so the discovery scoreboard and trade-archive page under-report realised P/L. A one-off, operator-triggered backfill recovers them from Binance trade history:

• POST /profiles/{profileId}/symbols/{symbol}/trade-archive-backfill (apps/api/src/routes/archive.ts) — acknowledges with 202 and enqueues a backfill-trade-archive pipeline job. An optional { from, to } ISO window bounds which round-trips are kept by their closing-fill time. Profile attribution is the operator's explicit (profile, symbol) choice: every fill on that symbol/account is credited to the chosen profile, so no clientOrderId inference is needed (myTrades does not carry it).
• apps/worker/src/queues/pipeline-handlers/backfill-trade-archive.ts — the handler. It paginates getMyTrades from fromId: 0 through the full history, then reconstructRoundTrips (a pure, I/O-free function in the sibling reconstruct-round-trips.ts) walks the fills oldest-first tracking running base quantity: a BUY adds, a SELL subtracts, and a round-trip closes when the position empties back to ~zero (a relative epsilon, default 1% of the cycle's peak position, absorbs base-asset fee dust). Each round-trip's gross profit, per-asset fees, and synthetic order summaries become a source='auto' row pinned to the closing-fill time. Synthetic order intents default to backfill. Exit-reason attribution is stricter and unrelated to the profile choice above: it first requires exactly one local row under the same account, profile, symbol, and Binance order id; that sole row must then be a closed FILLED SELL with a nonblank intent and a positive Decimal-equal executed quantity stored as a JSON string. Missing, malformed, or ambiguous proof stays backfill, which the UI honestly renders as unknown. Anything that cannot be priced honestly is dropped rather than emitted as bogus profit: an orphan sell (no matching prior BUY in the returned history), an overshoot cycle (sold more base than it bought, the surplus from a pre-history position), and a trailing open position. The first two are surfaced via a pipeline_backfill_trade_archive_uncosted_base warn with their counts.
• Idempotency is by closing-trade id and closing Binance order id: backfilled rows stamp their fills' tradeIds into the orders jsonb, and every archived row carries each order's binanceOrderId. A re-run skips any round-trip whose closing fill matches either marker, so re-running is a no-op. The order id is what catches overlap with forward-archived rows, which carry no trade ids because the forward archive summarises the local orders table. Both markers match the closing order only: a BUY whose partial fills straddle a flat-out legitimately appears in two cycles, and matching any shared order would discard the second real one. The { from, to } window bounds which round-trips are kept, not overlap.

Incomplete-history nudge Copy Section

The dev seed (scripts/seed-dev-data.ts) used to fabricate trade_archive rows with an empty orders array. A real archived round-trip always records its orders, so those seed rows misrepresented realised P/L on the archive page. Migration 0034_purge_fabricated_trade_archive_seeds.sql deletes them (source = 'manual' with an empty orders array). The seed writes archive rows again, but each one carries the two orders that produced it, so it can never match that predicate.

To surface genuinely-missing history, GET /profiles/{id}/trade-archive returns two lists, split by whether a backfill has been attempted. Both are omitted, never sent empty, on a ?view=rollup request — the dashboard's edge verdict and its live-vs-backtest card want only the by-source rollup, and running two whole-archive coverage scans to answer them was the bulk of a request they poll every 60s. An empty list is a claim that every coin is accounted for, which is what ends a running recovery, so a response that never computed the set has to say so by absence rather than assert it. A coin with fills but no archive is "missing history", but only running the backfill reveals whether that history is recoverable. The split avoids a dead-end nudge on coins that can never be rebuilt:

• recoverableSymbols — coins with a closed cycle (fills including at least one SELL) that no archive row accounts for, and no still-valid backfill attempt (per the backfill_attempts marker table, migration 0035). The actionable set: the archive page names each as a chip under a "Trade history incomplete" warning with a one-click Recover all. Coverage is tested per fill against the exchange order id, the one key both row shapes carry (forward rows summarise the local orders table, backfilled rows carry the ids reconstructed from myTrades). The cheaper "is this fill newer than the newest archived row" test is a per-symbol high-watermark and only ever finds a gap at the tail: let one cycle's archive job dead-letter and a later cycle archive after it, and the older gap sits below the watermark forever — invisible to the very sweep meant to repair it. Three further predicates keep the set honest. The SELL requirement excludes an open position — a coin the bot currently holds has fills and no archive row by definition, and nagging to "recover" a cycle that has not closed yet would name every held coin. And a marker only speaks for the fills that existed when it was written: a fill applied after attempted_at is history the attempt never saw, so the marker goes stale and the coin becomes actionable again. Without that check the live BTCUSDT timeline (BUY 2026-07-07, attempt 2026-07-11, SELL 2026-08-01) was invisible to both lists forever. attempted_at is the database clock read before the myTrades walk, not the write time at the end of it: a fill adopted during a multi-page walk is absent from that pass, and a write-time stamp would sort ahead of it and claim history the pass never saw. And the closing SELL must have settled (3 minutes): between its applied_fills commit and the forward archive's insert the cycle legitimately looks unarchived, and a sweep that enqueued a backfill in that window would race the forward path into a second P/L row for one cycle — the two paths derive cycle_end from different clocks (the execution report's event time vs the myTrades row time), so the partial unique index cannot collapse them. The grace applies only to this actionable list; unreconstructableSymbols enqueues nothing, so it has no race to lose and stays ungraced.
• unreconstructableSymbols — coins a backfill already tried and could not rebuild (no complete buy→sell cycle), each with a reason and a dismissed flag. Three reasons are derived from the reconstruct drop counts (overshoot / orphan-sells / open-or-pre-history, in that priority); the fourth, symbol-unavailable (backfill_attempts.symbol_unavailable, migration 0081), is not — the handler stamps it when the symbol is absent from a primed exchange-info cache, i.e. Binance no longer lists the coin, and it outranks the count-derived reasons because nothing could be read for that symbol at all, so "no closed cycle" would misdescribe why. A stale marker is excluded here too, so the two lists stay disjoint: a coin is never simultaneously "recover this" and "nothing to recover". Rendered as a quiet, non-actionable note (no recover button) so the coin explains itself instead of nagging. The operator can hide a coin from the note (POST /profiles/{id}/symbols/{symbol}/unreconstructable-dismiss with { dismissed }, persisted server-side as backfill_attempts.dismissed_at, migration 0036) and reveal it again via "Show hidden"; a re-attempt clears the flag.

Recover all fans out the per-symbol backfill (POST .../symbols/{symbol}/trade-archive-backfill) over recoverableSymbols and polls until that set drains: each coin either gains archive rows (recovered) or, via the marker the handler writes on every completion path, moves to unreconstructableSymbols. A coin that later trades a complete cycle is auto-archived forward, so it leaves both lists without re-attempt. The free-text "Recover a specific coin" fallback handles a coin traded entirely outside the bot (no applied_fills row).

The periodic sweep Copy Section

listRecoverableSymbols was originally read only by the archive screen, so a gap was repaired only if the operator happened to open the page and press the button — the same boot-only failure mode as the held-quantity backstop, and equally silent: nothing tells you that a trade is missing from your own history.

The archive-recovery-sweep cron (apps/worker/src/crons/archive-recovery-sweep.cron.ts) closes that loop. It runs the same profile-scoped query on a timer and enqueues the same backfill-trade-archive job the button does, so a cycle that closed without the forward archive firing heals on its own. selfReschedulePeriodMs: 900_000, not a pattern: the pass fans out one scoped query per active profile and each job it enqueues paginates myTrades, so a slow run must delay the next rather than overlap it and re-enqueue against the same account's Binance weight budget. Enqueues are capped at five symbols per profile per run, drawn from a window that rotates by five on every run. A fixed head would starve the tail: the query returns a stable alphabetical order and not every handler exit lands the attempt marker that drops a symbol out of the set (an unresolvable Binance client and a cold symbol-info cache both leave it recoverable), so five stuck symbols at the head would monopolise the budget forever. The rotation counter is in-process — the worker is single-replica, and a restart losing the count only re-picks the head. A per-profile failure is caught and retried next run, so one profile's fault cannot block every other profile's repair. Each enqueue carries a run-stamped jobId, never a static one: the pipeline queue retains terminal jobs, so a static id would be occupied forever after the first completion and every later repair silently dropped.

The loop is serial, so an unbounded await on one profile leaves every profile behind it unswept for as long as that query runs, and the tail log's {enqueued, deferred} could not tell a partially-reached list from a run that found nothing to repair. Each profile's query is therefore issued inside a transaction carrying a 30s statement_timeout (withStatementTimeout, packages/db/src/statement-timeout.ts), and every profile is counted by outcome into archive_recovery_sweep_profiles_total{outcome} (swept / failed / timeout) with the tail log widened to {active, swept, failed, timedOut, enqueued, deferred}. The bound is server-side on purpose: racing a timer against the promise would abandon the query, and pg returns a pooled connection only when its query settles, so an abandoned stall per 15-minute run would exhaust the 25-connection worker pool in about six hours and take ticks down with it. set_config('statement_timeout', …, true) is transaction-local, so it reverts at COMMIT or ROLLBACK and never caps the next borrower of that connection. The budget bounds a stall rather than policing a slow query: the healthy query returns in roughly a tenth of a second. Be careful what you credit it with, though. It is not what repaired the eight-hour run, and it is not what holds the cadence. The eight-hour run was a coverage subquery re-executing per candidate fill, and that was fixed in the query itself by binding every outer reference to its immediate parent so the planner flattens it into an anti-join, asserted by a plan-shape test; flattening is a rewrite-stage decision, so statistics drift cannot undo it. What the statement cap buys is structural: the loop is serial, so a statement that never returns parks the loop inside its await and no per-pass deadline can even be evaluated while that holds. A server-side cancel is the only mechanism that returns control without stranding the connection, which is what makes the pass budget below reachable at all.

The cadence is held by a second, separate bound: PASS_BUDGET_MS, two thirds of the period, checked before each profile is started. statement_timeout caps each STATEMENT and one profile issues two that can stall (the ownership join scopeProfile mints, then listRecoverableSymbols), so the per-statement cap alone would let a pass run for 2 x budget x active profiles with nothing bounding the profile count. When the pass budget is exhausted the run stops, counts the profiles it never reached into archive_recovery_sweep_profiles_total{outcome="unswept"}, and warns with {active, reached, unswept, budgetMs, elapsedMs}. Because the check happens between profiles rather than during one, a pass can overshoot by one profile's worst case of ~60s: 600s of budget plus 60s of overshoot stays inside the 900s period, so an exhausted pass still re-arms with a positive delay instead of back to back.

Stopping early would be a worse fault than the one it fixes if the pass always stopped in the same place, because listActive order is stable and the tail would then never be swept at all. So the handler keeps a resume cursor over the active-profile list, advanced by however many profiles a run reached, and the next run starts on the first profile the previous one did not — the starvation rotatedBatch prevents among one profile's symbols, one level up. Neither bound covers pool checkout — that is a third, separate bound: every pool is created with a 5s connectionTimeoutMillis, so a checkout against a saturated pool now fails instead of queueing forever (see Database → Pool checkout deadlines). For this cron that turns a saturated worker pool into a counted failed profile rather than a pass that never returns.

The manual Recover all remains, for when the operator wants the repair now rather than within the window.

Decision chain: failure semantics Copy Chapter

A tick's decisions are ORDERED on purpose — a cancel that clears the old resting order comes before the place that replaces it — so how the chain fails is what keeps Binance and the orders table agreeing. LiveExecutor.applyAll:

• Breaks on ANY failed order decision (place-order or cancel-order). After a failed cancel the old order is STILL RESTING on the exchange, so running the place behind it would leave two live orders while the local slot records one. After a failed place, the KV/event decisions behind it would run against a state that assumed the order landed.
• Still REPORTS every decision behind the break, stamped SKIPPED (phase: 'pre-call', retryable: true). Dropping them would read as "the strategy never emitted that order" — the exact wrong conclusion for the audit payload and for the override attribution. retryable: true because NOTHING WAS TRANSMITTED: re-issuing a skipped decision is provably safe, and that is what keeps both the strategy's own retry and an operator override alive when the SELL behind a transiently-failed cancel never got attempted.
• Assumes at most ONE place-order per tick. The retry is the un-advanced state (below), so the next tick re-emits the WHOLE array — two placements where only the second failed would re-place the first. Every strategy satisfies this; the executor logs an error if one ever does not.

The commit gate: the PLACEMENT's `phase` alone Copy Section

Whether the tick COMMITS nextState turns on the placement, and on its phase, never on retryable.

The placement, not the first failed order. The chain breaks on the first failed ORDER decision, which may be the CANCEL — and a cancel that dies on a transport error is ambiguous. Reading that would say "may be live ⇒ commit", while the SELL behind it was stamped SKIPPED and never transmitted at all. nextState hangs on whether the order it ASSUMED WAS PLACED was placed, so the gate reads the place-order decision's own result (applyAll enforces at most one per tick). The first failure survives only as the audit/alert attribution.

phase and retryable answer different questions and are orthogonal:

• phase — did the order execute? pre-call (refused before any HTTP call) and rejected (Binance parsed it and refused) are conclusive: it did NOT.
• retryable — would a retry ever succeed? A weight throttle drains; a -2010 insufficient balance does not.

So:

flowchart TD
    Result["place-order result"]:::io
    PhaseGate{"phase = pre-call or rejected?"}:::gate
    NoCommit["Do NOT commit nextState, order never landed"]:::wait
    Commit["phase accepted or ambiguous, commit nextState"]:::act

    Result --> PhaseGate
    PhaseGate -->|yes| NoCommit
    PhaseGate -->|no| Commit

    classDef io fill:#2c3e50,color:#ffffff;
    classDef core fill:#1f6f43,color:#ffffff;
    classDef gate fill:#f0ad4e,color:#2c3e50;
    classDef act fill:#1f6f43,color:#ffffff;
    classDef wait fill:#ecf0f1,color:#2c3e50;

Committing after a maybe-landed order (accepted / ambiguous) is what prevents a duplicate on retry: the next tick recomputes from an advanced state and does not re-emit the order that may already be live.

retryable survives only as the audit flag and the alert wording (willRetry). Note what it does NOT mean: willRetry: false says "the cause will not clear by itself", not "the bot has given up". The strategy still re-derives the order from its un-advanced state on every tick. The refusal circuit below decides how often that re-derived order is actually sent to Binance.

Repeated Binance refusals: three attempts, then one probe per minute Copy Section

A structural refusal is a parsed BinanceApiError with phase: 'rejected' and retryable: false. Three consecutive structural refusals trip a per-(account, profile, symbol) circuit only when both identities are byte-for-byte unchanged:

• request: clientOrderId, symbol, side, type, quantity, price, stop price, and time in force;
• rejection: Binance's numeric code and raw message.

Strategy metadata such as reason, override attribution, and whether an order is deferrable is deliberately excluded. Those fields do not change the request Binance judged. Conversely, the raw rejection message is retained because codes such as -2010 cover several distinct refusals.

Once tripped, the strategy still evaluates every market event and the audit still records its ordered decisions, but applyAll withholds the complete suffix from the first order onward until 60 seconds have elapsed. That includes a cancel before the placement; only a non-order prefix still runs. The existing multi-placement check runs first and still rejects the whole batch before any side effect. At the probe boundary, one real placement is allowed. The same refusal schedules the next probe for another 60 seconds; success, a transient or ambiguous failure, a changed request, or a changed rejection clears or restarts the sequence as appropriate.

The state is a 15-minute self-expiring Redis value under the profile and symbol namespace. Reads are part of the opening snapshot pipeline. A failed read fails open and skips circuit mutation for that tick; a failed write skips the matching condition update, so the durable diagnosis never claims a state Redis did not confirm. The tick's existing per-(profile, symbol) in-process chain serialises reads and writes, so this needs no Lua script and no distributed lock.

Tripping opens the degraded order-refusal-loop condition with the exact request and rejection in detail. Every known-state tick synchronises that condition, including suppressed ticks and the tick that clears it. The first two refusals use the normal order-failed notification. The trip and each refused probe use a separate alert path, throttled to one message per exact request-and-rejection identity per hour. Suppressed ticks do not alert or log; trips and actual probes do.

The retry IS the un-advanced state. There is no replay queue and there must not be one: a replayed order is a stale-priced order. Leaving the state un-advanced makes the next tick recompute from fresh market data and re-emit. The same predicate — (phase === 'pre-call' || phase === 'rejected') && retryable — is what the operator-override settle reads to decide whether to RE-ARM the override; the commit decision uses the phase half of it alone, because "safe to leave un-advanced" and "worth retrying" are not the same question.

A lost placement response (a transport throw: socket reset, timeout) is neither of the four phases until it is resolved. place-order probes Binance by its own clientOrderId, and the probe is only believed under two conditions: a "no such order" (-2013) is conclusive ONLY after Binance's recvWindow (plus a clock-skew margin) has elapsed since the request was SIGNED — until then the request can still be admitted — and a FOUND order counts as ours only if its time is at/after the send instant, since clientOrderIds are unique only among OPEN orders and strategies reuse a stable one per (profile, symbol).

Both anchors come from the client, not from the caller's wall clock, and that is load-bearing. The shared weight governor can hold a call for seconds before it signs (and a -1021 self-heal re-signs it after a /time round trip), so a deadline anchored to the pre-call reading expires while the request is still admissible. And order.time is stamped on BINANCE's clock, so the identity test compares against the send instant expressed in that clock (signedAtLocalMs + timeOffsetMs) — a raw local comparison accepts a namesake created in the skew window just before we sent. readSignedCallTiming(err) (@app/binance) carries both off the thrown transport error. Anything else stays ambiguous, which nothing ever retries. A false ambiguous costs one retry; a false accepted silently drops a protective stop; a false rejected duplicates a live order — so ambiguous is the only safe direction to err in.

The Binance ORDERS budget: block an exit, shed a reprice Copy Section

Binance meters order placement against an ORDERS budget that is separate from REQUEST_WEIGHT and scoped differently: weight is per-IP, ORDERS is per-account (UID). The budget is an unfilled order count — a placement adds one, a first fill subtracts one, and a cancel or an expiry changes nothing — so only placements are charged, and nothing is ever credited back. So the weight governor is shared process-wide while the order governor (packages/binance/src/rate-limit/order-governor.ts) is instantiated per (account, mode) — one bucket across N accounts would throttle each to 1/N of its real allowance.

Limits are read, never invented. They come from the rateLimitType: 'ORDERS' rows of exchangeInfo, which differ by environment (live spot publishes 100/10s and 200000/1d; the testnet 50/10s and 160000/1d), and ORDERS is enforced over all of those windows at once, so a reservation must fit in every one of them. parseOrderRateLimits skips any row whose interval it cannot map, and a governor built with no windows is inert — it admits everything. That is the deliberate posture when the limits cannot be read: no accounting is safer than accounting against numbers we made up. Each window's admission ceiling is floor(limit × targetUtilisation) (default 0.8, clamped to at least 1), the same haircut the weight governor already applies.

One charge point, one reconciliation point. binance-rest.ts already funnels every REST call through a single admission gate, so the ORDERS charge is one reserve(1) there and needs no per-callsite audit. It is gated on its own chargesOrderBudget argument rather than on the existing priority flag: priority covers cancels too, because a cancel-before-sell must not stall behind a bulk read cron, but a cancel spends no ORDERS budget. On the response, the governor reconciles itself against Binance's own x-mbx-order-count- headers. The header-name → window map is carried on the governor, produced by the same parseOrderRateLimits call that built its windows, so the two cannot desync. observe only ever tops up. A lower reported number has innocent causes — our own reservation is accounted before the request lands, Binance counts unfilled orders so a fill decrements theirs and not ours, and Binance's windows are fixed intervals that reset on a boundary while ours roll — but trusting any of them would let a burst double-spend our window. The rolling model is the conservative side of that trade: it admits at most the ceiling in any window-length span, where a fixed window admits up to twice that across a boundary.

Our tally is simultaneously an upper bound and a lower bound, which sounds contradictory until you name the frame. Against the flow this process placed itself it sits above Binance, because we never credit a fill back. Against Binance's true total it sits below, because orders placed in the Binance UI and a retried call that landed unseen are invisible to us. The first bias errs toward throttling early rather than toward a -1015; the second is exactly what observe exists to correct.

The block/shed split is the operator-visible part. reserve() blocks until there is headroom, which is right for an exit — getting out late beats not getting out — and wrong for re-pricing a resting protective stop, where the old order is still resting and still protective. So a placement can be marked deferrable on its intent, and applyAll peeks (hasHeadroom, which accounts nothing) before applying a batch that contains one. With no headroom, every ORDER decision in that batch is stamped DEFERRED (phase: 'pre-call', retryable: true — nothing was transmitted), the batch's KV and event decisions still run, and the tick records the order_budget_deferred metric plus one warn log naming the profile and symbol. DEFERRED also carries deferred: true, which is what keeps the operator's order-failed notification silent: a shed is a policy decision, not an outage, and alerting on the feature's designed steady state would teach the operator to ignore the channel that carries real failures. The decisions still appear in the tick's audit row, so the record is complete. The next tick re-derives the reprice from fresh prices, which is a better order than the one just skipped.

Three details make that safe:

• Peek, not reserve. Every order is accounted exactly once, at the REST admission point. The peek is therefore advisory: a concurrent reservation can consume the headroom between the peek and the call, which downgrades a shed into a short block — or, if what was lost was the last slot of a window too long to wait out, into the pre-call refusal below. Never into an over-admission, the only direction that would cost a -1015.
• Shedding the whole batch is safe because of the one-placement rule enforced immediately above it: a batch can never mix a deferrable reprice with an exit, so the only placement that can be shed is the deferrable one itself.
• Cancels shed with it, and must. The shed covers every place-order and cancel-order in the batch, even though a cancel spends no ORDERS budget. A re-price is the pair [cancel the resting stop, place the new one]; letting the cancel through while dropping the placement would retire a live protective order and leave the position naked until the next tick. Shedding both is what makes the re-price atomic — either it happens or nothing does.
• deferrable is generic. The executor keys on that boolean, never on a strategy's reason vocabulary, so the shed behaviour stays behind the plugin contract.

The wait is bounded. A non-deferrable placement blocks until there is headroom, and the wait is "until the blocking window's oldest record ages out" — for the 10s row at most ten seconds, but for the 1d row up to a whole day. Sleeping that out would be the worst available outcome: reserve runs inside the tick's per-(profile, symbol) chain, which is non-reentrant, so a parked reservation stalls every later tick for that symbol behind it — no reconcile, no stop re-price, no exit — with nothing surfaced to the operator.

So a cumulative wait longer than MAX_RESERVE_WAIT_MS (60s), measured across every re-check inside one reserve call, throws OrderBudgetUnavailableError instead. Cumulative rather than per-wait because the short rows would otherwise escape the bound entirely: a 10s window can never produce a single wait over 60s, so a per-wait check would leave it looping while an operator placing orders by hand on the same UID keeps it saturated — and with no queue and no fairness, a waiter that keeps losing the re-check to a sibling profile has nothing to break it.

That error has its own type for one reason: place-order classifies it ahead of the transport-error branch, as phase: 'pre-call', retryable: true. The distinction is load-bearing — the refusal happens before the request is signed, so the order provably never reached Binance, and probing for an order that was never sent could only ever resolve as ambiguous, which is unretryable and would strand the placement over a rate limit that clears by itself. The tick ends, the failure is visible on the normal alerting path, and the next tick re-derives the order against a window that has since decayed.

Saying plainly what saturation costs. Once a window is that saturated, reserve refuses every placement it gates, exits included — it takes no deferrable argument, and the throw precedes any notion of priority. That is deliberate, not an oversight: admitting past our own ceiling would make the governor decorative and walk into the -1015 it exists to avoid. Three things bound the damage. The refusal is retryable and alarmed, so it is a delayed exit, not a lost one. A position with protectiveStop enabled still has its STOP_LOSS_LIMIT resting at Binance, which needs no budget to trigger. And reaching the daily ceiling at all should take roughly 128 000 placements (testnet's 160 000 row after the 0.8 haircut) — far above any rate the strategies sustain, since each bounds its own demand. The 10s row is the one that can realistically bind, and it clears in ten seconds. The bound is there for the case where that estimate is wrong.

Waiting instead of shedding would hold the (profile, symbol) chain lock and delay the next tick's exit check — strictly worse than a late reprice. The residual risk this absorbs is a burst rather than a sustained rate: minute boundaries are globally synchronised, so held symbols re-arm in the same second. A strategy bounds its own sustained demand (momentum's profitTrail.ratchetMinutes and protectiveStop.minRearmDriftPct); the governor is what turns the burst into "a reprice arrived a tick late" instead of a -1015 on the call that mattered most.

Duplicate-MARKET guard: durable placement dedup Copy Section

A MARKET order fills and closes instantly, freeing its deterministic clientOrderId — Binance dedupes only OPEN orders — so a re-emitted identical MARKET order is accepted and fills AGAIN. The state-layer read-your-writes CAS is the primary guard; executor/placement-dedup backstops it. record-on-accept remembers the placed clientOrderId and seenRecently suppresses a same-id MARKET placement within a 60s window; a SELL calls forgetSymbol so a legitimate re-entry (same stable entry id) is never suppressed.

The dedup has a durable per-symbol Redis SET mirror (placement-dedup:<symbolKey>) behind the in-process Map. It is a self-expiring idempotency mirror, NOT a lock (no owner, no release; the SADD+PEXPIRE is one atomic MULTI/EXEC so the SET can never exist without a TTL, and always self-heals). It exists for the killed-pod case below: a BullMQ retry on a fresh pod has an empty Map but still reads the prior MARKET placement from Redis. The read fails open (a stalled or failed lookup allows the placement through, leaving the Map + state-layer CAS as primary), and the SET enforces a COARSER, key-level window than the Map — every record re-stamps the whole key's TTL, and SISMEMBER ignores a member's own age — so an old id can read as "seen" past its own window; that is the safe direction (over-suppress never double-fills).

SIGTERM: teardown gated on a clean drain Copy Section

SIGTERM starts a bounded drain (stop the lifecycle components and the BullMQ queueSet, capped at a deadline). Destructive teardown is now GATED on that drain completing cleanly. A drain that times out OR whose queueSet.closeAll() rejects skips pool.end() / redis.quit() and exits non-zero, so the orchestrator SIGKILLs the process against a HEALTHY pool. Closing the pool out from under an in-flight tick would flip it to ending and poison that tick's state commit into a BullMQ retry — and that retry, re-deriving a MARKET entry on a fresh pod, is exactly what the durable placement-dedup mirror above guards against double-filling.

Reconciling orders whose profile is gone Copy Chapter

Deleting a profile DETACHES its orders (profile_id → NULL) rather than deleting them: a resting order is real money on Binance and its ledger row must outlive the strategy. Two paths close such a row once the exchange says the order has left the book:

• The user-data stream — the event-router routes a detached order's terminal executionReport to fillAdopter.reconcileDetachedFill (ledger-only: no cost basis, no strategy state, no re-subscription — there is no profile left to adopt into). It passes the exchange's own eventTimeMs, so closed_at is the same whichever path settles the row.
• The detached-orders-reconcile cron (every 10 min, offset from orphan-orders-detect so the two sweeps do not contend for the same account's Binance weight) — driven off the ORDERS TABLE, not the active-profile set. It exists for the case that CREATES detached orders in the first place: deleting an account's LAST profile detaches its orders AND tears down the only stream the account had, so no report will ever arrive again and every profile-driven sweep is structurally blind to it. The account's key pair outlives its profiles, which is enough to ask Binance for each order's true status. A -2013 (the exchange never knew the id) closes the row CANCELED; any other failure leaves it LIVE for the next tick — never a guessed terminal status for an order that may be resting.

Without either, the row stays closed_at NULL forever: permanent phantom exposure that counts toward countAccountOpenExposure and so blocks the very account delete that created it.

Operator overrides: settling on the outcome Copy Chapter

An override (force-buy, force-sell, manual order) is written to a Redis key with a 300 s TTL and an override_actions row. The tick's bundle-builder reads the key with a non-destructive GET and then DELs it before tick() runs, so the override is at-most-once by construction and the worker owes the operator an honest answer about what became of it. The read and the delete are two commands on purpose: pipeline() is not transactional, so an in-pipeline GETDEL would consume the override even when a sibling slot errored, and a server-side delete cannot be undone.

Arming replaces the override before it. There is one Redis key per (profile, symbol) and arming SETs it blindly, so a second force-sell on a symbol overwrites the first before any tick can read it: only the NEWEST override can ever run. record therefore settles the row it replaces superseded in the SAME transaction as the insert. Left unconsumed, that predecessor is hidden behind the newer row while it lasts — GET /override serves only the newest row in the window — but it outlives it. Cancel the replacement and the ghost becomes the newest row: an override that reads as still pending though its key was overwritten and no tick can ever run it, until the stranded-row sweep relabels it. One transaction because the two halves are only true together: settled without the insert loses the operator's override entirely.

This is best-effort against a concurrent arm rather than an enforced invariant. The transaction runs at READ COMMITTED, so neither arm can see the other's uncommitted insert: whichever predecessor already existed is settled at most once, and both NEW rows survive — degrading to the ghost above, not to a lost override. Serialising them would need an advisory lock for a race only a double-tapping operator can reach; the sweep already cleans up after it.

Bounded to rows that are still PENDING and carry a symbol. A row with processing_at or picked_up_at set is left alone, for the reasons in the two sections below; an account-wide dust conversion (symbol is null) has no Redis key to overwrite, so two queued conversions are two real pieces of work rather than a replacement. If the enqueue after the insert fails, writeOverrideAndEnqueue compare-and-deletes the Redis key it wrote and runOverrideOrRollbackDb settles the NEW row rejected. If that Redis delete itself fails it raises OverrideRollbackError and the new row is left PENDING on purpose, because the override may still be live. Either way the predecessor stays superseded rather than being restored: once the arming SET lands its key is gone and it is unexecutable from that moment. In the narrower window where the SET itself fails, the predecessor's key survives its TTL, but the settled row makes the claim fail, so the tick stands down and nothing is dispatched.

That answer is derived from the ORDER, never from the strategy's intent to act:

1. The strategy stamps intent.overrideActionId on any order it emits for the override. The worker correlates by id equality only — it holds no strategy vocabulary, so this stays generic (invariant 1).

2. The daily-loss breaker filter (applyDailyHalt) returns what it SUPPRESSED as well as what it kept. A suppressed order never reaches the executor, so without that set a breaker-killed override would look like a strategy that simply chose not to act.

3. settleOverride then either re-arms the Redis key or settles the row with a typed outcome on override_actions.outcome (applied / rejected / unknown / superseded / expired), which GET /override serves and the SPA polls after its 202. Re-arm iff (phase === 'pre-call' || phase === 'rejected') && retryable — see docs/architecture/extensibility.md for the phase/retryable split. An ambiguous failure (5xx, transport error, unparsable error body) is NEVER re-armed and notifies the operator: the order may already be live, and only a human can check. phase is decided inside @app/binance at throw time (BinanceApiError.phase), by the only code that can see whether Binance's error body actually parsed; the worker reads it, never re-derives it.

   The settle is deadline-bounded and the unknown notification is fire-and-forget: the tick runs inside the per-(profile, symbol) chain lock, so an unbounded await there delays the NEXT tick for that symbol. The settled row is the durable record.

   A tick that placed ANY order never re-arms, even if the strategy also signalled a defer. That coarse guard is a deliberate fail-safe on top of the exact overrideActionId attribution, not a duplicate of it: a plugin that places the override's order but forgets to stamp the id would otherwise look like a tick that emitted nothing, and the re-arm would produce a SECOND live market order.

4. Every one of those answers assumes the tick got far enough to give one. It may not: it can throw (a schema-invalid bundle, a strategy fault, an executor fault) or it can RETURN early on a self-healing skip (the weight governor's bulk-read backpressure, raised by the candle load downstream of the bundle read). Either way the key is already gone and nothing is left to retry it. So the tick carries an override ticket: the assembler arms it the instant the builder hands over a consumed override, the settle disarms it, and a finally — which covers both exits — compensates whatever is left. That runs INSIDE the chain lock: released first, the next tick could consume a newer override and this one's re-arm would write the stale intent back over an empty key.

   The compensation goes through the same settleOverride, with an aborted fate, and re-arms only when a retry is neither dangerous nor futile:

   • dispatched an order → never re-armed; settles unknown and notifies, because the order may have reached Binance and only a human can resolve that.
   • the tick died at the strategy boundary (a throw from bundleSchema.parse or strategy.tick) → never re-armed; settles rejected. The cause may be the operator's payload or assembler drift in any other bundle slot, and the worker cannot tell which — but it does not need to, because tick() is pure and the next tick would die at the same line either way. Re-arming would loop it to the TTL while the symbol commits no state at all, leaving a live position with no trailing sell and no protective stop. This is the abort fate's futility test, the counterpart of supported for a defer and retryable for a failed order.
   • the operator cancelled it meanwhile → never re-armed. The claim below covers the dispatch window only, and a re-arm releases it, so the row is deletable again either side of that: the operator can still revoke in the gap between the consuming DEL and the compensation. Before re-arming, the ticket re-reads the live row and stands down unless it is still the same overrideActionId — including when that read cannot be confirmed. Losing an override is recoverable (the operator re-presses); executing a revoked force-sell is not. One Postgres read, only on an abort that consumed an override, so the warm path is untouched.
   • no window left to restore (elapsed, or the builder surfaced no TTL) → settles expired.
   • otherwise → re-armed for the next tick, the I/O faults that are genuinely transient.

   It never throws, so the error that failed the tick is still the one that reaches the DLQ.

Claiming the row: the cancel race. That GET-then-DEL makes the override at-most-once, but it says nothing about the operator changing their mind. DELETE /override deletes an UNCLAIMED row (deletePendingForSymbol carries processing_at is null), and that guard is the only thing standing between a cancel and a dispatch already in flight. So the tick claims the row, and the claim is the arbiter both ways:

• When. overrideTicket.arm() fires claimAction the instant the builder hands over a consumed override — the earliest the race can be closed — so the round-trip overlaps strategy.tick() instead of sitting in front of the order.
• The gate. The tick awaits the claim immediately before it marks itself about to dispatch, and a claim it did not win stands the WHOLE tick down as a graceful skip: no order, no state commit, no audit. The next tick re-derives its decisions from fresh state. Aborting everything rather than just the override's own orders is both simpler and safer — those decisions were computed from a bundle carrying an override that turned out not to be ours.
• The deadline runs from the gate, not from the claim. The round-trip is started at arm but left unbounded there, and the persist budget is applied where the dispatch actually blocks on the answer. Bounded at arm instead, that budget would be spent on the tick body in between (candle windows, which can fall back to a weight-governed REST call, plus tick() itself), so any tick slower than about 100 ms would discard a claim it already held and stand down for nothing. Total tick time stays bounded at tick body plus one budget, so the chain lock is still never held open indefinitely.
• Fail closed, in two distinguishable ways. A CAS that returns false is lost: the operator's cancel demonstrably won. A claim that throws or misses its deadline is unresolved: nobody can say. Both bar the dispatch, because while the row is unclaimed the cancel route will delete it, and dispatching under an unconfirmed claim can put an order on the exchange for an action the operator was told was cancelled.
• Standing down is not losing the intent — unless nobody can tell who owns the row. The gate returns before markOrderAttempted(), so compensation still sees orderAttempted === false and takes its usual route: re-arm for the REMAINING window when the operator's intent is still there, or stand down without an outcome when the row is gone (they already have their answer). An unresolved claim is the exception and settles rejected instead: a database that cannot answer this claim inside the budget will not answer the retry's either, so re-arming would consume-and-re-arm every tick until the window drained, and the symbol would commit no state and place no orders the whole time — no trailing sell, no protective stop, on a live position. The same futility test markDeterministicAbort applies to a poison payload.
• Release before re-arm, and FENCED. settleOverride releases the claim before restoring the Redis key, and outside the try around that SET: a claimed row is neither cancellable by the operator nor claimable by the tick that picks the key back up, and a release fault must cost at most a stuck claim (the stale-claim reaper clears it) rather than the operator's intent. The release is not gated on believing the claim is held — a lost acknowledgement is exactly when that belief is wrong, and exactly what would livelock the retry — and what makes that safe is the fence: the tick supplies the processing_at stamp to claimAction and passes the same value to releaseClaim, which matches on equality. raceDeadline abandons a write rather than cancelling it, so an unfenced release could land minutes later and strip a LIVE claim off a different tick's dispatch, taking the cancel guard off with it.
• A claimed row is exempt from the supersede above, and that exemption is what keeps this 409 reachable. Settling a claimed row hides it from findActiveForSymbol, which is the read the cancel route uses to decide between 204 and 409 — so arming a second override while the first was mid-dispatch would make the route answer "cancelled" about an order on the wire, the exact failure this 409 exists to prevent. Claimed-A plus pending-B therefore remains a legitimate pair: two rows for one symbol are both "active", and cancelling B while A runs is the half-measure the 409's wording already owns.
• The operator's side. A cancel that meets a claimed row gets 409 CONFLICT naming the action as in flight, and the Redis key is NOT evicted (the tick is still reading it). A pending row still cancels: row deleted, key evicted, 204. So does a symbol with no active row at all — key evicted, 204 — which is also the orphan-key cleanup. The bulk symbol wipe is unchanged at 204: it is a wholesale teardown, not a request about that one action, and the claimed row it cannot delete is left to the worker's settle and the sweeps.
• A worker that dies holding a claim self-heals: reapStaleProcessing (dust cron, every 5 min, claims older than 10 min) nulls processing_at, and reapExpiredForAccount settles the row terminally once it is older than the outcome window (10 min), on the next 5-minute cron pass. settle carries no processing_at predicate, so a claimed row settles like any other and the claim can never block the operator's answer. That reaper runs ABOVE both of the cron's gates, credential resolution and the live-only check: it is pure Postgres and needs no REST client, while overrides are armed on testnet exactly as on live, so behind either gate an abandoned claim would never be cleared by anything. The cancel route also refuses to be held hostage by one — past OVERRIDE_CLAIM_STALE_MS (10 min), the same horizon the reaper uses, so the API can never call a claim dead before the reaper has had its chance, it evicts the orphaned key and answers 204 rather than 409 forever, while leaving the row to the reaper instead of clearing another consumer's claim from the API.

The breaker wins. A BUY-side override submitted while the daily-loss halt is armed is refused at the API with 409 before any row or Redis key is written — the halt runs to the next UTC day and the override lives five minutes, so accepting it would only produce a delayed rejection. That 409 is a fast-fail UX shortcut, not the enforcement point: the API keys off the REQUEST's side, the WORKER off the side of the decision the strategy actually emits, so the worker is the backstop. Both state the refusal with the same DAILY_ENTRY_HALT_REASON constant from @app/contracts, so the api's 409 body and the worker's recorded rejection reason cannot drift. If the breaker arms in the gap between the 202 and the tick, the tick settles the row rejected with the breaker's reason and does not re-arm it (re-arming is provably futile). Exits (trigger-sell, force-eject, cancel-order, a SELL-side manual order) are never gated: the breaker pauses new risk, it never traps the operator in a position.

Stranded rows. An override whose re-arm failed, or whose worker was killed outright mid-tick, would sit pending forever — the ticket above handles a tick that throws or skips, but not one that never gets to run its finally. The dust-snapshot sweep (below) settles any symbol-scoped row older than the outcome window, and it runs before that cron's demo / testnet skips, because overrides are armed in those modes too.

Those two ways of stranding need opposite advice, so the tick leaves a pick-up breadcrumb: override_actions.picked_up_at (migration 0075). It is stamped once, at the moment the tick marks itself as about to put an order on the wire — the same place-order gate that bars a re-arm from then on — and awaited before applyAll dispatches anything. Not at consume time: a tick that consumes an override and then emits no order (the strategy declined, the operator cancelled inside the window, a re-arm whose window later lapsed) has provably dispatched nothing, and stamping it would alarm the operator about a live order that never existed while destroying the true "nothing ran, press it again" reading of that row. A cancel-order-only override does not stamp either, matching the re-arm gate's own scoping: nothing can be left resting. The sweep is therefore two statements per account, and the branch that reads the breadcrumb runs FIRST:

A breadcrumbed row is also exempt from the supersede that arming performs, and for the same reason the sweep splits these two branches at all: unknown is the only outcome that raises the override-unresolved notification, a settled row is immutable, so an early superseded would suppress the one alert that sends a human to the exchange — and it would say "replaced before it ran" about an order that may be resting there.

The notification is the only operator-visible surface for a swept row, which is why it exists rather than being a nicety. The sweep's staleness bound is now - OVERRIDE_OUTCOME_WINDOW_MS on created_at <, and GET /override reads created_at >= that same constant: same 600 000 ms, opposite comparisons, so the two sets are disjoint. A row this sweep settles can never be served to the SPA, and the expired … Try again copy in the web outcome hook is unreachable for it (that copy renders only for in-tick expiry, where trying again IS the right advice). The row itself remains the audit record.

The expired branch stays deliberately silent: nothing was placed, so there is nothing to check. The unknown branch raises one notification per row, naming the symbol — the same category the tick raises for an ambiguous outcome of its own, because from the operator's side it is one problem. Breadcrumb-set rows are settled first so that a stamp landing between the two statements is simply missed by this sweep and labelled unknown by the next one; NULL-first would already have recorded expired — "nothing ran" — terminally, about an order that may be resting on the exchange, and with no notification to correct it.

The breadcrumb is deliberately not processing_at: a claim is a lease that releaseClaim and the stale-claim reaper both null, so it cannot hold a fact that has to survive the very crash that summons those reapers — see database.md. It would also break cancellation, since a cancel skips a claimed row.

Two gaps, and only one of them is harmless.

A crash before the stamp. The DEL is not transactional with Postgres, so a worker can die between consuming the key and stamping. That row is swept expired, which is the truth: the stamp is triggered by the same marker that says "about to dispatch" and is awaited before applyAll, so nothing can have been dispatched inside this window.

A stamp that fails or misses its deadline. Different, and a real hole. raceDeadline resolves rather than rejects, so a failed or timed-out stamp still lets the dispatch proceed; if the worker then dies, a live order gets an un-breadcrumbed row and the sweep calls it expired — "no tick ran", with no notification. That is the optimistic direction, not a conservative one. It is the deliberate price of never failing an operator's force-sell on a diagnostic write, and it is not silent: both modes warn with the row's attribution, which is the only trace of it.

Dust conversion to BNB Copy Chapter

"Dust" is a leftover balance too small to trade — a few cents of a base asset stranded after a sell or shaved off by fees. Binance's "Convert Small Balance to BNB" SAPI (POST /sapi/v1/asset/dust-btc to list, POST /sapi/v1/asset/dust to convert) sweeps it into BNB once per 6-hour window, live accounts only. Conversion is operator-initiated only — the bot never moves funds automatically. The earlier auto-convert policy cron was removed (its safety filter excluded only base-asset prefixes, never quote/funding suffixes like USDT, so it could sweep the operator's funding balance); the whole feature — cron, dust_auto_convert_config table, and its API toggle — is gone.

• dust-snapshot.cron.ts (every 5 min). Per active live profile, refreshes the cached dust-eligible set (dustEligible Redis key the GET /profiles/{id}/dust-transfer route serves) and executes any pending dust-transfer override_actions via convertDust under a claim → convert → finalise lifecycle (stale-claim reaper, partial-conversion warn, retry-next-tick). This is the only caller of convertDust. On finalisation it stores Binance's convert response on the action row (override_actions.result) as durable history and fires the account-scoped dust-transfer notification (a money-path alert), so a conversion is never silent.
• Operator surface. The manual /accounts/{accountId}/dust-transfer screen lists the eligible set and enqueues a dust-transfer override action (POST /profiles/{id}/dust-transfer, triggeredBy='user'); a slow Binance call runs off the request thread. The screen also shows recent conversions via GET /profiles/{id}/dust-transfer/history, reading each action's requested assets and stored result (converted assets + BNB received). BNB (the destination) and BTC (the valuation quote) are excluded as non-source assets on the screen.

Frame-trace record/replay correctness gate Copy Chapter

The strategy golden-fixture replay (quality gate 5) replays a hand-built candle series through strategy.tick. The worker frame-trace gate complements it from the other end: it replays a recorded worker frame — the verbatim Redis blobs, the live-price override, the trigger, and the resolved profile slice that drove one real tick — through the REAL buildTickInput + the REAL strategy.tick. It proves two distinct things:

• The assembly path stays correct across refactors: snapshot-loader key ordering, account/open-order parsing, indicator revival, and the livePrice → currentPrice override. The override is asserted directly at the assembler boundary (built.input.market.currentPrice === tuple.livePrice on a live-price frame), not inferred from decisions — see the caveat below.
• Strategy decision-drift = 0 for the assembled-input → tick path: the strategy's emitted decisions deep-equal the recorded decisions.

Two scope limits to read honestly, so the gate is not overclaimed:

• The momentum tuples derive decisions only from candlesByInterval (closed-candle EMA cross) and never read market.currentPrice, so by themselves they cannot prove the live-price override is decision-stable — a regression that broke the override would leave momentum's decisions identical. That is exactly why the override is also checked directly at the assembler boundary. To make the decision-drift assertion bite a real currentPrice-driven branch, the fixture additionally carries a trailing-trade tuple whose livePrice (47000) is the only thing that crosses its stop trigger (avgEntryPrice 50000 × stopLossPercentage 0.97 = 48500): with the override the strategy emits a grid-stop-loss MARKET SELL place-order; without it currentPrice would be the flat closed-candle close (50000) and the decision would be noop. So a regression that broke the override or the stop-loss math now surfaces as decision drift, not just an assembler-boundary mismatch.
• The per-(profile, symbol) state slice is fed verbatim: StatePort.loadForTick is stubbed to return the recorded state unchanged, so the state reconcile/migration path is not exercised here. State migration is covered by its own unit suite.

Recording. Set WORKER_FRAME_TRACE=1 (optionally WORKER_FRAME_TRACE_FILE=; the default is worker-frame-trace.jsonl under the OS temp dir, never inside the repo tree, since the trace holds account balances and holdings) and run the worker. createFrameRecorderFromEnv (apps/worker/src/tick/frame-recorder.ts) returns a JSONL file-sink recorder; otherwise it returns undefined and the tick handler's if (deps.frameRecorder) guard is a true no-op (single branch, no other cost). After each successful strategy.tick, the handler appends one tuple per line: { raw, livePrice, trigger, intervals, profile, decisions }. The raw blobs are appended verbatim (already JSON strings from Redis), so re-serialising can never introduce drift.

Fixture. Committed tuples live in apps/worker/__tests__/tick/fixtures/frame-replay/sample.jsonl — at least three: two momentum frames (including one mini-ticker live-price frame whose livePrice is a finite positive string distinct from the closed-candle close) and one trailing-trade frame that emits a non-noop live-price-driven place-order. The momentum tuples are authored to be internally consistent; the trailing-trade tuple's decisions are machine-generated from the real strategy.tick (see Regenerating below) rather than hand-written, so they cannot silently diverge from strategy output.

Regenerating the fixture. The trailing-trade tuple is produced by apps/worker/__tests__/tick/frame-replay.generate.test.ts, which builds the tuple inputs, runs the real strategy.tick through the same deterministic harness the replay test uses (_frame-replay-harness.ts, so record-time inputs == replay-time inputs), and pins the result to the committed line as a golden drift-guard. After a legitimate strategy change that alters this decision, the guard fails loudly; run bun --filter @app/worker test:frame-replay:generate, review the new decision, and recommit the regenerated sample.jsonl line. The generator writes its trace only to a tmpdir, never the repo tree.

Replay test. apps/worker/__tests__/tick/frame-replay.test.ts loads the fixture, builds a fake in-memory Redis whose pipeline returns each tuple's blobs in snapshot-loader key order, stubs StatePort / coldLoad / marketDataPort / symbolInfoCache deterministically, resolves the real strategy from the registry by the recorded strategyName, drives the real buildTickInput + strategy.tick, and asserts output.decisions deep-equals the recorded decisions. It runs without Postgres or testcontainers.

Run it. bun --filter @app/worker test:frame-replay (also wired into scripts/ci/test-replay.sh alongside the strategy replays).