Self-healing broker clients, per-firm scheduler isolation, eager boot

Why the autotrader sometimes broke after a weekend until `pm2 restart`:
`next start` only logs uncaught errors, so nothing ever crashed, and a few
paths could leave a client dead for the life of the process.

Tradovate: a closed socket scheduled exactly one re-login, with no catch —
one failure during weekend maintenance (or a rate-limit penalty returned as
a 200 without a token) and the client never came back. connectAndAuth is now
a single-flight loop that never throws: jittered backoff capped at 5 min,
penalty tickets waited out, captcha demands backed off. A 15s watchdog
abandons silent, never-authorized or stuck-closing sockets and stale syncs;
renewal falls back to a fresh login; orders fail fast (OrderNotPlacedError)
on a dead socket and time out instead of hanging; a bad sync answer no
longer wipes accounts and positions.

TopstepX: hub start/stop/invoke and every REST call are bounded; a wedged
hub is discarded and rebuilt; a stuck sync is superseded after 3 min;
re-login no longer tears down the sync loop; a watchdog re-logs when syncs
go stale; a no-fill timeout cancels the order.

Scheduler: one unready client used to hold every firm forever. It now holds
the tick for a 10-minute grace, then trades the other firms without it.
Per-firm try/catch in batch collection, one batch at a time, accounts marked
in flight, legs un-counted only when the entry never reached the broker, and
the scheduler starts before the first batch fires.

Boot and supervision: instrumentation.ts boots the process at server start
(it used to wait for the first dashboard request); a supervisor rebuilds a
client that has been unhealthy for 20 min (not a TopstepX one holding a
position); broker HTTP calls default to a 60s timeout; GET /api/health shows
each client's liveness and the scheduler's last decision. Shared state the
eager boot would otherwise split across bundles lives on globalThis. Logs no
longer carry access tokens or whole AxiosErrors (which include the encoded
Tradovate password).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

app/api/health/route.ts

@@ -0,0 +1,72 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { NextResponse } from 'next/server';
+import { getClients } from '@/lib/clients';
+import { getTradableFirms } from '@/lib/firm-catalog';
+import { getSchedulerHealth, getSchedulerStatus } from '@/lib/auto-trade';
+import { getBuildVersion } from '@/lib/version';
+import { errorMessage, isClientReady } from '@/lib/platform-client';
+
+// Liveness has to be computed now, never served from a cache.
+export const dynamic = 'force-dynamic';
+
+/** A one-line marker file, or null when it isn't there. */
+function readMarker(...segments: string[]): string | null {
+    try {
+        return readFileSync(path.join(process.cwd(), ...segments), 'utf8').trim() || null;
+    } catch {
+        return null;
+    }
+}
+
+/**
+ * GET /api/health — is this instance able to trade, and if not, why.
+ *
+ * The numbers to read before restarting a broken box, and what start-all
+ * prints after boot. Always 200 when the handler runs: `ok` in the body is the
+ * verdict (every tradable firm that has a client is synced, and synced
+ * recently), so a caller can tell "unhealthy" from "not answering".
+ *
+ * Calls getClients(), so hitting it also boots an instance that somehow
+ * hasn't been.
+ *
+ * Built field by field on purpose — firm rows carry passwords and API keys,
+ * and nothing here may echo a credential or a token.
+ */
+export async function GET() {
+    try {
+        const now = Date.now();
+        const clients = getClients();
+
+        const entries = getTradableFirms().map((firm) => {
+            const base = { firmId: firm.id, firm: firm.name, platform: firm.platform };
+            const client = clients.get(firm.id);
+            if (!client) return { ...base, hasClient: false, ready: false, accounts: 0 };
+
+            const ready = isClientReady(client, now);
+            const accounts = client.accountList.length;
+            try {
+                return { ...base, hasClient: true, ready, accounts, ...client.health() };
+            } catch (err) {
+                // One client's broken health() must not hide the rest.
+                return { ...base, hasClient: true, ready, accounts, healthError: errorMessage(err) };
+            }
+        });
+
+        return NextResponse.json({
+            ok: entries.every((e) => !e.hasClient || e.ready),
+            now,
+            uptimeSec: Math.round(process.uptime()),
+            version: getBuildVersion(),
+            // Read from disk on every call: the build .next holds now (the
+            // running one, unless a build landed since this process started),
+            // and the commit the updater last finished building.
+            buildId: readMarker('.next', 'BUILD_ID'),
+            builtCommit: readMarker('scripts', '.built-commit'),
+            scheduler: { ...getSchedulerStatus(), ...getSchedulerHealth() },
+            clients: entries,
+        });
+    } catch (err) {
+        return NextResponse.json({ ok: false, error: errorMessage(err) }, { status: 500 });
+    }
+}