senofy/autofirmer-expanded
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Self-healing broker clients, per-firm scheduler isolation, eager boot

Browse Source 
Why the autotrader sometimes broke after a weekend until `pm2 restart`:
`next start` only logs uncaught errors, so nothing ever crashed, and a few
paths could leave a client dead for the life of the process.

Tradovate: a closed socket scheduled exactly one re-login, with no catch —
one failure during weekend maintenance (or a rate-limit penalty returned as
a 200 without a token) and the client never came back. connectAndAuth is now
a single-flight loop that never throws: jittered backoff capped at 5 min,
penalty tickets waited out, captcha demands backed off. A 15s watchdog
abandons silent, never-authorized or stuck-closing sockets and stale syncs;
renewal falls back to a fresh login; orders fail fast (OrderNotPlacedError)
on a dead socket and time out instead of hanging; a bad sync answer no
longer wipes accounts and positions.

TopstepX: hub start/stop/invoke and every REST call are bounded; a wedged
hub is discarded and rebuilt; a stuck sync is superseded after 3 min;
re-login no longer tears down the sync loop; a watchdog re-logs when syncs
go stale; a no-fill timeout cancels the order.

Scheduler: one unready client used to hold every firm forever. It now holds
the tick for a 10-minute grace, then trades the other firms without it.
Per-firm try/catch in batch collection, one batch at a time, accounts marked
in flight, legs un-counted only when the entry never reached the broker, and
the scheduler starts before the first batch fires.

Boot and supervision: instrumentation.ts boots the process at server start
(it used to wait for the first dashboard request); a supervisor rebuilds a
client that has been unhealthy for 20 min (not a TopstepX one holding a
position); broker HTTP calls default to a 60s timeout; GET /api/health shows
each client's liveness and the scheduler's last decision. Shared state the
eager boot would otherwise split across bundles lives on globalThis. Logs no
longer carry access tokens or whole AxiosErrors (which include the encoded
Tradovate password).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Li
2026-09-23 20:18:33 -05:00
co-authored by Claude Opus 5.5
parent 6146031a38
commit b932452e98
13 changed files with 2793 additions and 627 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/clients.ts
+253 -15
View File
@@ -1,23 +1,32 @@
import { TradovateClient } from './tradovate-class';
import { TopstepXClient } from './topstepx-class';
import type { PlatformClient } from './platform-client';
import { errorMessage, isClientReady, type PlatformClient } from './platform-client';
import { getInstruments } from './db';
import { getTradableFirms, startCatalogSync } from './firm-catalog';
import { resolveFirmAssignments } from './account-type-resolver';
import { resolveContracts } from './contract-resolver';
import { startReporter, sendReport } from './reporter';
import { startBackupSync } from './backup-sync';
import { applyBrokerProxy } from './outbound-proxy';
import { applyBrokerHttpDefaults, applyBrokerProxy } from './outbound-proxy';
import { startMasterLink } from './master-link';
import { resumeSchedulerIfPersisted } from './auto-trade';
// Use global to persist the client pool across HMR reloads in dev mode.
// Keyed by the master's firm id, which is a uuid string.
//
// It is also what keeps the pool single in production: instrumentation.ts
// boots it from its own server bundle, and the route handlers can each hold a
// separate instance of this module. Registry state therefore lives here, never
// in module-level variables.
const g = global as typeof globalThis & {
__tradovateClients?: Map<string, PlatformClient>;
__tradovateClientsInitialized?: boolean;
__tradovateClientCreds?: Map<string, string>;
__contractResolverTimer?: ReturnType<typeof setInterval>;
__contractResolveRetryTimer?: ReturnType<typeof setTimeout>;
__contractResolveRetries?: number;
__clientSupervisorTimer?: ReturnType<typeof setInterval>;
__clientSupervision?: Map<string, SupervisionEntry>;
};
function ensureMap(): Map<string, PlatformClient> {
@@ -74,7 +83,14 @@ export function initClient(
export function removeClient(id: string): void {
const client = ensureMap().get(id);
client?.disconnect();
// A throwing disconnect() must still take the client out of the pool:
// left in, half torn down, it would never reconnect — and the supervisor
// rebuilding it would fail the same way every time.
try {
client?.disconnect();
} catch (err) {
console.error(`[clients] disconnect failed for firm ${id}: ${errorMessage(err)}`);
}
ensureMap().delete(id);
credsMap().delete(id);
}
@@ -127,11 +143,21 @@ export function reconcileClients(): void {
export function getClients(): Map<string, PlatformClient> {
const map = ensureMap();
// `next build` imports route modules to collect page data. Nothing there
// should call this, but a build that did must never dial the brokers or
// resume the scheduler — so hand back the empty pool and leave the flag
// unset, as if nothing had happened.
if (process.env.NEXT_PHASE === 'phase-production-build') return map;
if (!g.__tradovateClientsInitialized) {
g.__tradovateClientsInitialized = true;
try {
// Before anything dials out: put the operator's proxy (if any)
// under the broker clients. Master traffic stays direct by design.
// Before anything dials out: a timeout on every broker HTTP call,
// and the operator's proxy (if any) under the broker clients.
// Master traffic stays direct by design. (outbound-proxy applies
// both when it loads, too; these are idempotent.)
applyBrokerHttpDefaults();
applyBrokerProxy();
// Pull the catalog before anything else — without it there are no
@@ -158,6 +184,9 @@ export function getClients(): Map<string, PlatformClient> {
// Schedule daily resolve at midnight
scheduleDailyResolve();
// Rebuild any client that has been wedged for too long.
startClientSupervisor();
// Start master dashboard reporter
startReporter();
@@ -180,30 +209,67 @@ export function getClients(): Map<string, PlatformClient> {
// ── Contract auto-resolve ────────────────────────────────────────────────────
// With no authenticated Tradovate client yet — a cold catalog, a slow login,
// broker maintenance at boot — keep checking once a minute for half an hour
// instead of giving up until midnight. Missing it leaves the contract cache
// cold: the settings page shows nothing, TopstepX loses the resolver's
// front-month names, and the first trade pays for the lookups itself.
const CONTRACT_RESOLVE_RETRY_MS = 60_000;
const CONTRACT_RESOLVE_MAX_RETRIES = 30;
function triggerContractResolve(): void {
// A midnight run supersedes a pending retry; either way only one chain.
if (g.__contractResolveRetryTimer) clearTimeout(g.__contractResolveRetryTimer);
g.__contractResolveRetryTimer = undefined;
const map = ensureMap();
// Find first client with a valid access token (the Yahoo-driven resolver
// confirms contract ids against Tradovate; TopstepX clients resolve their
// own contracts but still reuse the resolver's front-month names).
// A token alone isn't enough: a client whose session died keeps its
// expired one, and resolving with that fails with nothing left to retry.
let accessToken: string | null = null;
for (const [, c] of map) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const token = (c as any).accessInfo?.accessToken;
if (token) { accessToken = token; break; }
if (!token) continue;
let authenticated = true;
try { authenticated = c.health().authenticated; } catch { /* no health: trust the token */ }
if (authenticated) { accessToken = token; break; }
}
if (!accessToken) {
console.log('[contract-resolver] No authenticated Tradovate client yet — skipping resolve');
const retries = (g.__contractResolveRetries ?? 0) + 1;
if (retries > CONTRACT_RESOLVE_MAX_RETRIES) {
console.log(
`[contract-resolver] No authenticated Tradovate client after ${CONTRACT_RESOLVE_MAX_RETRIES} retries`
+ ' — giving up until the midnight resolve',
);
g.__contractResolveRetries = 0;
return;
}
if (retries === 1) {
console.log('[contract-resolver] No authenticated Tradovate client yet — retrying every 60s');
}
g.__contractResolveRetries = retries;
g.__contractResolveRetryTimer = setTimeout(() => triggerContractResolve(), CONTRACT_RESOLVE_RETRY_MS);
return;
}
g.__contractResolveRetries = 0;
const symbols = getInstruments().filter((i) => i.enabled).map((i) => i.symbol);
console.log(`[contract-resolver] Resolving ${symbols.length} instruments...`);
resolveContracts(symbols, accessToken)
.then((results) => {
const rolled = Object.entries(results).filter(([, v]) => v?.alternative);
console.log(`[contract-resolver] Done — ${rolled.length} contract(s) rolled via price match`);
})
.catch((err) => console.error('[contract-resolver] Resolve failed:', err));
// Runs from timers: a throw here (the instruments read) would escape as
// an uncaught exception.
try {
const symbols = getInstruments().filter((i) => i.enabled).map((i) => i.symbol);
console.log(`[contract-resolver] Resolving ${symbols.length} instruments...`);
resolveContracts(symbols, accessToken)
.then((results) => {
const rolled = Object.entries(results).filter(([, v]) => v?.alternative);
console.log(`[contract-resolver] Done — ${rolled.length} contract(s) rolled via price match`);
})
.catch((err) => console.error(`[contract-resolver] Resolve failed: ${errorMessage(err)}`));
} catch (err) {
console.error(`[contract-resolver] Resolve failed: ${errorMessage(err)}`);
}
}
function scheduleDailyResolve(): void {
@@ -215,7 +281,179 @@ function scheduleDailyResolve(): void {
const now = new Date();
if (now.getHours() === 0 && now.getMinutes() === 0) {
console.log('[contract-resolver] Midnight resolve triggered');
// Its own half hour of retries, not what's left of a chain it cuts off.
g.__contractResolveRetries = 0;
triggerContractResolve();
}
}, 60_000);
}
// ── Client supervision ───────────────────────────────────────────────────────
/**
* Rebuilds a client that has been unable to trade for a long time.
*
* The clients reconnect themselves, but only along the paths their authors
* foresaw. A re-login that fails in an unexpected way, an await that never
* settles, a socket that dies without a close event — any of those used to
* leave a client wedged for the life of the process, and because the
* scheduler waits on every firm, one wedged firm stopped them all. After a
* weekend of broker maintenance that meant a pm2 restart. A fresh client is
* exactly what the restart gave it, so the registry hands one out itself:
* same firm, same credentials, new instance.
*
* "Unready" is isClientReady() saying no — the same rule the scheduler and
* /api/health use: not synced, or no good sync in the last 5 minutes.
*
* Deliberately slow — 20 minutes unready before a rebuild, and at most one
* rebuild per firm per 20 minutes. The clients' own retries get the first go,
* and a broker that is really down, or rate-limiting logins, must not be
* hammered with new sessions. Some clients are never replaced while unready
* (see rebuildHold): one whose own login retry is pending, and a TopstepX
* one holding a position.
*/
const SUPERVISOR_INTERVAL_MS = 60_000;
const REBUILD_AFTER_UNREADY_MS = 20 * 60_000;
const REBUILD_COOLDOWN_MS = 20 * 60_000;
/** The message TopstepXClient records when a firm has an API key but no
* username. That client never even tries to log in, so a rebuild changes
* nothing — adding the username changes the credentials, and
* reconcileClients rebuilds it then. */
const TOPSTEPX_USERNAME_MISSING = /username missing/i;
interface SupervisionEntry {
/** The instance being judged. A different one (rebuilt here, or replaced
* by reconcileClients after a credentials change) starts a fresh clock. */
client: PlatformClient;
/** When it was first seen unready (0 = ready at the last check). */
unhealthySince: number;
lastRebuildAt: number;
/** Last time a due rebuild was held back and said so (log rate limit). */
holdLoggedAt: number;
}
function supervisionMap(): Map<string, SupervisionEntry> {
if (!g.__clientSupervision) g.__clientSupervision = new Map();
return g.__clientSupervision;
}
function startClientSupervisor(): void {
// HMR-safe, and safe when resetClients() re-runs the bootstrap.
if (g.__clientSupervisorTimer) clearInterval(g.__clientSupervisorTimer);
g.__clientSupervisorTimer = setInterval(superviseClients, SUPERVISOR_INTERVAL_MS);
}
/** Why a client isn't ready, short enough for one log line. Never the token. */
function describeUnready(client: PlatformClient, now: number): string {
const parts: string[] = [];
try {
const health = client.health();
parts.push(`transport ${health.transport}`);
if (health.lastLoginError) parts.push(`login error: ${health.lastLoginError.slice(0, 160)}`);
} catch {
parts.push('health unavailable');
}
parts.push(client.lastSyncAt > 0
? `last good sync ${Math.round((now - client.lastSyncAt) / 60_000)}m ago`
: 'never synced');
return parts.join(', ');
}
/**
* Why replacing this client now would do more harm than good, or null.
*
* - Its login keeps failing and its own retry is already scheduled. Nothing
* in the instance is broken — the broker is down, refusing the
* credentials, or rate-limiting — and a fresh instance would fail the same
* way while starting its backoff over at seconds: a burst of logins after
* every rebuild instead of one every few minutes, which is how a login
* rate-limit turns into a captcha lockout.
* - TopstepX with a position open. The gateway has no OCO, so the client
* itself cancels the surviving exit leg when the other fills; a fresh
* instance knows nothing of those pairs, and the survivor would be left
* working at the broker after the position closes — an order that can open
* a position nobody asked for. It waits until the account is flat.
*/
function rebuildHold(platform: string, client: PlatformClient, now: number): string | null {
try {
const health = client.health();
if (health.consecutiveLoginFailures > 0 && health.nextReconnectAt > now) {
return `its login retry is due in ${Math.max(1, Math.round((health.nextReconnectAt - now) / 60_000))}m`
+ ` (${health.consecutiveLoginFailures} failure(s) in a row)`;
}
} catch { /* no health: judge it on the rest */ }
if (platform === 'topstepx' && Object.keys(client.positions ?? {}).length > 0) {
return 'it holds an open position (TopstepX exit brackets are tracked only inside the client)';
}
return null;
}
function superviseClients(): void {
// Runs inside setInterval: nothing may escape, or the process logs an
// uncaught exception and the supervisor keeps firing into the same fault.
try {
const map = ensureMap();
const book = supervisionMap();
const now = Date.now();
// A firm that left the registry takes its bookkeeping with it.
for (const id of [...book.keys()]) {
if (!map.has(id)) book.delete(id);
}
let firms: ReturnType<typeof getTradableFirms> | null = null;
for (const [id, client] of [...map]) {
try {
let entry = book.get(id);
if (!entry) {
entry = { client, unhealthySince: 0, lastRebuildAt: 0, holdLoggedAt: 0 };
book.set(id, entry);
} else if (entry.client !== client) {
entry.client = client;
entry.unhealthySince = 0;
}
if (isClientReady(client, now)) {
entry.unhealthySince = 0;
continue;
}
if (!entry.unhealthySince) {
entry.unhealthySince = now;
continue;
}
if (now - entry.unhealthySince < REBUILD_AFTER_UNREADY_MS) continue;
if (now - entry.lastRebuildAt < REBUILD_COOLDOWN_MS) continue;
// Only read when a rebuild is actually due — it hits the DB.
firms ??= getTradableFirms();
const firm = firms.find((f) => f.id === id);
if (!firm) continue; // no longer tradable: reconcileClients drops it
if (firm.platform === 'topstepx'
&& TOPSTEPX_USERNAME_MISSING.test(client.lastFetchErrors?.['login'] ?? '')) continue;
const minutes = Math.round((now - entry.unhealthySince) / 60_000);
const hold = rebuildHold(firm.platform, client, now);
if (hold) {
if (now - entry.holdLoggedAt >= REBUILD_COOLDOWN_MS) {
entry.holdLoggedAt = now;
console.log(`[clients] ${firm.name} unhealthy for ${minutes}m — not rebuilding while ${hold}`);
}
continue;
}
console.log(
`[clients] ${firm.name} unhealthy for ${minutes}m (${describeUnready(client, now)})`
+ ' — rebuilding client',
);
entry.lastRebuildAt = now;
removeClient(id);
entry.client = initClient(firm.id, firm.name, firm.platform, firm);
entry.unhealthySince = now;
} catch (err) {
console.error(`[clients] supervisor check failed for firm ${id}: ${errorMessage(err)}`);
}
}
} catch (err) {
console.error(`[clients] supervisor tick failed: ${errorMessage(err)}`);
}
}