Why the autotrader sometimes broke after a weekend until `pm2 restart`:
`next start` only logs uncaught errors, so nothing ever crashed, and a few
paths could leave a client dead for the life of the process.
Tradovate: a closed socket scheduled exactly one re-login, with no catch —
one failure during weekend maintenance (or a rate-limit penalty returned as
a 200 without a token) and the client never came back. connectAndAuth is now
a single-flight loop that never throws: jittered backoff capped at 5 min,
penalty tickets waited out, captcha demands backed off. A 15s watchdog
abandons silent, never-authorized or stuck-closing sockets and stale syncs;
renewal falls back to a fresh login; orders fail fast (OrderNotPlacedError)
on a dead socket and time out instead of hanging; a bad sync answer no
longer wipes accounts and positions.
TopstepX: hub start/stop/invoke and every REST call are bounded; a wedged
hub is discarded and rebuilt; a stuck sync is superseded after 3 min;
re-login no longer tears down the sync loop; a watchdog re-logs when syncs
go stale; a no-fill timeout cancels the order.
Scheduler: one unready client used to hold every firm forever. It now holds
the tick for a 10-minute grace, then trades the other firms without it.
Per-firm try/catch in batch collection, one batch at a time, accounts marked
in flight, legs un-counted only when the entry never reached the broker, and
the scheduler starts before the first batch fires.
Boot and supervision: instrumentation.ts boots the process at server start
(it used to wait for the first dashboard request); a supervisor rebuilds a
client that has been unhealthy for 20 min (not a TopstepX one holding a
position); broker HTTP calls default to a 60s timeout; GET /api/health shows
each client's liveness and the scheduler's last decision. Shared state the
eager boot would otherwise split across bundles lives on globalThis. Logs no
longer carry access tokens or whole AxiosErrors (which include the encoded
Tradovate password).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PM2 runs update-check.mjs as a cron *restart*, and PM2 7 stops a process
that is still running at the next minute boundary. Any update that needed
npm install or next build (more than ~55s) was killed half-way: the lock
was left behind, HEAD had already moved at the pull so the next run saw
nothing to do, and next build had already wiped .next under the live
server — a box that broke after a weekend push and stayed broken.
- The cron run is now a quick CHECK: fetch, decide, and hand the work to a
detached worker (--apply) that PM2's next tick can't reach.
- "Up to date" is judged against scripts/.built-commit (what .next was built
from), not HEAD, so a pull that was never built finishes on the next check.
First run on a machine adopts a complete build that is newer than HEAD's
last move and rebuilds anything else — that rescues boxes the old updater
already stranded.
- The previous build is set aside as .next-prev and restored if the build
fails, so a broken push never leaves a box without a build; failures retry
after 15 min, then hourly (scripts/.update-failed).
- The lock holds a pid and is atomic; build dirs are deleted without ever
following the links in .next/node_modules into the real node_modules.
- Every instance in scripts/instances.json is restarted, and python/pip
children stay hidden on Windows now that the worker has no console.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`node -` puts '-' in argv[1], so the registry snippets in add-instance.sh
and remove-instance.sh were one slot off: they wrote the registry to a file
named "-" in the working directory, with the instances.json path as the
instance name, a null port and the port as the db. scripts/instances.json
was never created, so the updater and start-all only ever knew instance 1 —
after an update, instance 2 and up kept running old code against the new
build until someone ran `pm2 restart all`.
Devices that already ran add-instance.sh need their scripts/instances.json
written by hand (and the stray "-" file deleted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a fund transaction starts a new cycle (the days revert to 0), the
cycle it ended is checked against the completion rule for a week after and
named if it stood complete — its final day may never have been reported.
Only for a cycle this instance saw begin: the clients now note where the
ended cycle started as they move last_fund_date forward. Completions carry
the cycle they belong to, and each firm reports its whole account list so
the master can tell erased accounts and spot new funded ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cycleCompletion() says whether today's P&L finished the current cycle
(profit target, consistency and minimum days all met) — projectPerfectPath's
completion probe run without and with today. The reporter counts those as
`completed` (a subset of targetHit) and sends their names, so the master
can keep an account a firm erases after it passes. Checked ahead of the
lockout branch: a firm locking a passed account is no DLL loss. The local
dashboard shows them with a purple Completed pill.
Because the master keeps a completion for good, doubtful reads stay
'none': weekends, a Monday with a weekend-dated row, the first minutes
after the 5 PM roll, and history that hasn't freshly loaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/db.ts switched to WAL at module load before setting busy_timeout.
next build imports it from several workers while the live server holds
the same file, and a journal-mode change needs the file to itself — so
the pragma failed instantly with SQLITE_BUSY and took the whole build
down ("Failed to collect page data"), blocking updates on any box whose
database was still in rollback mode. The timeout is now set first and
the WAL switch is wrapped: a held file keeps its current mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fleet tooling is npm-only (setup-linux.sh installs with npm,
update-check.mjs triggers installs off package-lock.json), so the
committed yarn.lock did nothing for any VPS — but yarn regenerating it
locally dirtied working trees and blocked git pull on machines where
yarn had ever been run. Now gitignored: yarn dev still works wherever
someone wants it, with a local lockfile git never sees.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matches the master's tokenless-push change: pushBackup no longer waits
for master_backup_token — every instance with a uuid backs up on its
own, and the 5-minute tick force-pushes so a poisoned master copy
self-heals. The token now only gates pulling a backup back (restore),
so setup-linux.sh asks for it only when restoring and the Settings
help text says so.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four features:
- Backup: lib/backup-sync.ts pushes the full settings table and firm
credentials (the deliberate exception to "credentials never leave the
machine") to the master's bearer-gated /api/instances/backup — every
5 minutes when the content hash changed, within seconds of a settings
or credential edit, and on the new backup-now command. Enabled by the
master_backup_token setting.
- Restore: a dead VPS's replacement pulls the backup and adopts its
uuid/name, so roster, payouts and card links carry on. Pre-boot via
seed-settings.js --restore-from (setup-linux.sh prompts for it) or
the Settings page's restore picker (lib/backup-restore.ts). Stop the
old machine first; the master 409s a live-name rename.
- Proxy: the proxy_url setting routes BROKER traffic only — TopstepX
REST + SignalR hub, Tradovate REST + websocket, Yahoo quotes — via
lib/outbound-proxy.ts (axios agents, undici dispatcher, ws agent).
The master link, reporter, catalog and backup push stay direct, so a
dead proxy cannot take down the control channel. Restart to apply.
- Multi-instance: AUTOFIRMER_DB moves the SQLite file so one checkout
runs N instances; scripts/instances.json (per-device, gitignored)
drives start-all/update-check, and add-instance.sh / remove-instance.sh
manage the extra PM2 processes. Absent registry = single-instance
behavior, unchanged. Instance 1 keeps port 3000 for the clicker and
extension.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A drain stage zeroed consistency, so the payout was planned (and the
account stopped trading) as soon as the min days were booked — even when
the cycle's biggest day still exceeded the firm's ratio (Topstep holds
its 40% through the drain). Carry stage.consistency into the effective
config and keep booking drain days until the cycle total covers
maxDay / consistency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a popup button that scans the Topstep dashboard's GraphQL API
(crystal.topstep.com) for passed evaluations that can still be
activated into an Express Funded account.
An eval is activatable when its status is "Passed" and its subscription
id is not shared by ANY funded account it ever spawned — regardless of
that funded account's current status, since activation is one-way. The
result and its progress are mirrored to chrome.storage so the popup
renders them independent of the message port (avoids the "message port
closed" error on this ~20s scan). Activatable accounts link to their
Topstep detail page.
Adds https://*.topstep.com/* host permission; bumps to 0.15.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the single hardcoded signed-out check with a guards list on each
firm: any guard can trip mid-run via a URL substring (checked on every
step) or a selector probe (checked when a step fails). Each guard defines
recovery steps or a human-wait fallback, a per-run trigger cap, and a
resume policy — 'continue' retries the tripped step, 'restart' rewinds
the run to step 1.
Runner bumped to 0.20.0; the server still ships the legacy
signedOut/authSteps shape for stale runners, and a new runner synthesizes
a guard from a legacy payload.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both the MLL floor and the books-invariant fund detector assumed a $0
starting balance (correct for Express, wrong for funded/eval sizes). A
150K account (accountSize 150000) built its balance curve from $0, so
the floor came out near -$4,500 and a $145k balance never crossed it —
the account never marked dead — while reconcileFunds saw the whole
$150k as an unexplained deposit. Seed both with accountSize so the floor
is an absolute balance (150K/$4,500 drawdown → 145,500, locked at the
funded size once the EOD high climbs a drawdown above it) and the books
invariant nets out the starting balance. Skip reconcile until the
account type is known, since the start balance is unknowable before then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
projectPerfectPath worked on the raw daily P&L, so for a combining firm a
holiday-session trade (e.g. the Labor Day session) counted as its own
trading day — an account one real day short of its min days looked
complete and anchored its payout to the last traded day. The projection
now folds sessions through combineCmeSessions like the live path, and
projected days skip dates that clear into the next full day.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A completed cycle's payout now lands on its last traded day (requestable
after that session's close) instead of whatever day the projection ran —
a Friday finish viewed over the weekend stayed pinned to Sunday. Also
stop inserting the platform's stale weekend realizedPnL as a Sat/Sun
P&L row, which fabricated an extra qualifying day and made accounts one
day short look already complete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MLL floor no longer recomputes while a position is open or while
the balance disagrees with the books beyond fee noise — the last
settled floor stands, so balance ≤ floor can only come true when a
booked fill crosses a settled floor (Topstep's own liquidation fill
included). The floor also survives canTrade flipping off, so a blown
account stays ☠ instead of demoting to inactive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
23 deterministic values per (account, date) instead of $5/$10/$15; the
qualifying floor moves to the grid minimum so every draw counts toward
the cycle. The per-stage min-day field no longer drives the amount.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TopstepX never reports the Maximum Loss Limit, but it's deterministic:
floor = min(0, eodBalanceHigh − trailingDrawdown), locking at the $0
starting balance. The EOD curve rebuilds from daily P&L + fund events,
with off-book fee drift spread across days proportionally to reported
fees (validated against a platform-reported MLL to ~$0.10). Stored in
autoLiqThresholds — negative, or +$0.01 once locked — so dead detection,
dashboards, and the ruin model's startThreshold all use the real room.
Guards move from '> 0' to '!== 0'; 0 still means no floor. canTrade=false
accounts are skipped. daily_pnl gains a fees column for the weighting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@microsoft/signalr does an unconditional require('ws')/require('eventsource')
when constructing a connection; bundled (turbopack) builds throw 'dynamic
usage of require is not supported', which the login chain mislabelled as a
login failure and retried every 60s. Externalize the package and never let
hub setup fail the login — REST polling covers everything the hub does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-linux.sh accepted being run as root and simply set SUDO="". That quietly
broke everything user-scoped: PM2 keeps its process list in $HOME/.pm2 and the
login entry goes in $HOME/.config/autostart, so under sudo both landed in /root.
The install reported success, then `pm2 list` as the real user showed nothing -
it was talking to a different daemon - and the autostart entry sat in a home
directory that never logs in.
Both scripts now refuse, and name the account to use when SUDO_USER says how
they were invoked. setup-linux.sh still uses sudo for apt, which is the only
part that needs it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A payoutable account with no withdrawal stages (e.g. MFFU same-equity)
now withdraws its share and re-targets the same equity each cycle instead
of stopping after the first payout. Non-payoutable evals still stop at
one completion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When target and min trading days are already satisfied, the payout fires
on the current trade date instead of being pushed to the next session —
so an account that has already hit its target with its days in shows the
payout as requestable now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An eval that completes its target passes to the next account rather than
paying out — the projected completion marker reads 'Passed ✓' unless the
type is marked payoutable on the master.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The version is the checkout's HEAD commit (short hash + commit time),
read once at startup — updates arrive as `git pull`, so there is no
release number to bump. It rides on the socket hello, socket reports,
and the HTTP report fallback; the field is optional in the protocol so
older masters ignore it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- projectPerfectPath books each cycle's payout ON its final trading day
and anchors ruin probabilities to the platform's real liquidation floor
(Tradovate auto-liq) when reported, else the configured trailing
drawdown with the maxLoss/$2,000 fallback
- Account calendar shows the cumulative reach % on every projected day,
caps the projection at two payouts ahead, and renders potential days
as dashed outlines instead of shaded cells
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
projectPerfectPath prices every projected withdrawal as
min(withdrawalPct × equity profit, cap) × profit split, and tracks the
cumulative probability of reaching it under the fair-walk ruin model —
room/(room + dayTarget) per day (the per-leg product telescopes), trailing
threshold locked at breakeven after the first withdrawal, no obfuscation
branches. The reporter attaches the first 6 payouts per account so the
master's payout calendar renders real amounts and probabilities instead of
one static expectedPayout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- TopstepXClient (ProjectX Gateway): API-key auth, SignalR user hub,
Trade/search daily P&L, emulated OCO exits, books-invariant
withdrawal detection for $0-start accounts
- Platform-aware client pool, credentials (api_key column), and
credential form; $0-size account types match by prefix alone
- Drain stages: min-day ($5-$15 randomized) MNQ legs for the
configured days, then wait for the 50% withdrawal; stage pills
show 'Stage N – Drain'
- Account calendar projects the perfect path: future days book their
planned targets, simulated 50% withdrawals, payout marker
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stop-enabled accounts show an amber "stop $X" beside the day target:
the day-1 net floor (live against realized P&L) or 2x a min-day's
target, mirroring exactly what the trade path will place — and nothing
when the next trade rides stopless.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stop-enabled account whose day booked red via its own stop shows an
amber "Personal DLL" badge on the dashboard (the firm's lockout keeps
its red "Hit DLL") and reports as personalDll in the day outcomes
instead of loss, per account type and per firm.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When an account type opts in, legs are placed as an OCO pair via
order/placeoco - the usual limit plus a stop: the first cycle day's stop
keeps the day's net floor at -$1,000, min-day trades stop at 2x their
target, and the day after any red day rides stopless to make it back. A
stop-out books the day: eligibility refuses the account once realized
P&L falls short of the legs fired (or is simply negative on an opted-in
account, which survives restarts). If placeoco is rejected the leg falls
back to the classic lone limit with a loud log rather than leave a
position without an exit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Big days (> the $1,700 split threshold) must carve at least the account
type's minLegs from the catalog — pickLegAmount reserves the minimum
piece for each leg still owed, forcing a carve even once the residual
dips below the threshold, clamped to what the pieces allow. Days at or
below the threshold still trade as one leg. The smallest split piece
drops from $800 to $500. An older master without the field means no
minimum.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sessions roll at 5 PM CT, so after tonight's open a day booked on
tomorrow's trade date is already in the day count — lastDay is what lets
the master tell those accounts apart from ones that have not traded the
current session yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
payoutStates: one entry per live account (negative included, dead not)
with its current-cycle progress — qualifying days, full equity profit,
max qualifying day, P&L sum, and the cycle's end-of-day high-water mark —
plus the withdrawal count and the cycle's effective target/consistency/
min-days from resolveEffectiveConfig. client.dailyPnL is already the
active-cycle slice, so a withdrawal resets the reported path on its own.
Feeds the master's payout calendar; an older master ignores the field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The payout calendar projects accounts still on the winning path, so the
daysHist now skips accounts below their starting balance or flipped to
liquidate-only, alongside the dead ones the report already drops. They
still count in the slice totals — only the calendar stops seeing them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds daysHist ({daysTraded: accountCount}) to AccountTypeStats, computed
from the same combined-day view the trade path uses and covering every
live account including ones mid-trade. Feeds the master's payout
calendar; an older master simply ignores the field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An account Tradovate flips to liquidate-only at the daily loss limit is
done for the day, but the reporter only counted a loss when the booked
realizedPnL was already negative — so a locked-out account whose P&L
hadn't caught up (or that had no matching config) showed as still to
trade. Count inactive accounts as traded at a loss.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Like accounts in a batch previously carved legs with independent
Math.random, so near-identical day targets could split very differently.
Leg fractions are now drawn from groupLegFraction(prefix, tradeDate,
legIndex) — FNV-1a seeded like dailyBump — so accounts of the same
account type on the same leg of the same day split identically when
their remaining targets match, and proportionally close otherwise.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-account wipe works one account at a time; when every account is
carrying stale cached rows (the pre-roll dates again), that's a lot of
clicking. POST /api/accounts/wipe-all drops the cached rows for every
account on every firm and rebuilds them all from fresh reports — firms
in parallel, accounts within a firm at fetchDaysTraded's own pacing.
The toolbar button confirms inline with the same 28-day-window warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The daily target sometimes inflates to base + consistency×todayPnL (e.g.
1200 + 0.4×1218.48 = 1687.39), which happens when a session's cached
daily-P&L row is dated something other than the current tradeDate: it
then lands in BOTH the prior-day list and realizedPnL, so day-start
equity reads too low while the prior-day sum stays high.
Detect it with a book-balance invariant that is coincidence-proof (it
does not care whether prior and today P&L happen to be equal): the
reconstructed day-start equity (equityProfit − realizedPnL) must equal
priorProfit + sum(prior days). The state route now logs the full account
state whenever those disagree by more than $50, and GET /api/debug
reports the same per-account breakdown on demand (?account=<substring>)
so the exact inputs can be captured the moment a number looks wrong.
Read-only; no behavior change. Remove once the trigger is fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cached daily P&L rows outlive what the Cash History report still
produces (stale pre-roll dates being the recent example), and nothing
deletes them any more. POST /api/accounts/[id]/wipe drops the account's
cached P&L, fund transactions and meta, clears the in-memory view, and
rebuilds from a fresh report — fetchDaysTraded now takes an optional
account filter so one account doesn't cost a report round-trip on every
other. The button confirms inline first: rows older than the report's
28-day window cannot be refetched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cash History rows can carry a weekend calendar date (Sunday-evening
activity attributed pre-roll), and those slipped through the holiday
combining as their own trading day — accounts traded only across the
Labor Day Sun/Mon/Tue stretch read 2/5 instead of 1/5. Weekends never
clear at CME any more than holidays do, so treat Saturday and Sunday as
unsettled dates and roll them forward with the rest of the stretch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
