PM2's cron for autofirmer-update dies inside a live daemon (six Windows
machines in October), and nothing on the machine noticed. The app now
watches scripts/.last-check and, past 10 minutes, runs
update-check.mjs --heal: a checker PM2 still lists is restarted (which
re-registers its cron), one it lost is registered again, and one whose
errored entry still holds a live pid is left alone, since PM2 would kill
that pid. The heal scrubs PM2's per-app fields from its environment first.
The master's new 'update-now' command runs --heal --now, skipping a
failed update's retry wait. A failed pull, npm install or build records
one line of reason (or the chown to run when a file isn't ours), which
rides on .last-check to the master. A package-lock.json rewritten by npm
is put back before the pull. The fetch timeout drops under the cron's
minute. scripts/.no-auto-update opts a checkout out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PM2 runs update-check.mjs as a cron *restart*, and PM2 7 stops a process
that is still running at the next minute boundary. Any update that needed
npm install or next build (more than ~55s) was killed half-way: the lock
was left behind, HEAD had already moved at the pull so the next run saw
nothing to do, and next build had already wiped .next under the live
server — a box that broke after a weekend push and stayed broken.
- The cron run is now a quick CHECK: fetch, decide, and hand the work to a
detached worker (--apply) that PM2's next tick can't reach.
- "Up to date" is judged against scripts/.built-commit (what .next was built
from), not HEAD, so a pull that was never built finishes on the next check.
First run on a machine adopts a complete build that is newer than HEAD's
last move and rebuilds anything else — that rescues boxes the old updater
already stranded.
- The previous build is set aside as .next-prev and restored if the build
fails, so a broken push never leaves a box without a build; failures retry
after 15 min, then hourly (scripts/.update-failed).
- The lock holds a pid and is atomic; build dirs are deleted without ever
following the links in .next/node_modules into the real node_modules.
- Every instance in scripts/instances.json is restarted, and python/pip
children stay hidden on Windows now that the worker has no console.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four features:
- Backup: lib/backup-sync.ts pushes the full settings table and firm
credentials (the deliberate exception to "credentials never leave the
machine") to the master's bearer-gated /api/instances/backup — every
5 minutes when the content hash changed, within seconds of a settings
or credential edit, and on the new backup-now command. Enabled by the
master_backup_token setting.
- Restore: a dead VPS's replacement pulls the backup and adopts its
uuid/name, so roster, payouts and card links carry on. Pre-boot via
seed-settings.js --restore-from (setup-linux.sh prompts for it) or
the Settings page's restore picker (lib/backup-restore.ts). Stop the
old machine first; the master 409s a live-name rename.
- Proxy: the proxy_url setting routes BROKER traffic only — TopstepX
REST + SignalR hub, Tradovate REST + websocket, Yahoo quotes — via
lib/outbound-proxy.ts (axios agents, undici dispatcher, ws agent).
The master link, reporter, catalog and backup push stay direct, so a
dead proxy cannot take down the control channel. Restart to apply.
- Multi-instance: AUTOFIRMER_DB moves the SQLite file so one checkout
runs N instances; scripts/instances.json (per-device, gitignored)
drives start-all/update-check, and add-instance.sh / remove-instance.sh
manage the extra PM2 processes. Absent registry = single-instance
behavior, unchanged. Instance 1 keeps port 3000 for the clicker and
extension.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was no way to tell "running, nothing to pull" from "not running". The
no-change path logs nothing by design, and PM2 reports a cron-restart process
as `stopped` with ↺ 0 even while firing on schedule — I verified that against
PM2 7.0.4: a one-minute cron fired four times without the counter moving once.
scripts/.last-check is now rewritten on every run with the outcome. It is a
single overwritten line, so it needs no trimming.
The outcome is set by each exit path and written once in `finally`, rather than
calling record at each return — the first version of this did the latter and
already missed the build-failure path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Capped at 256 KB, trimmed back to the newest 500 lines. Done once per run
before anything is written: the no-change path logs nothing, so that is the
only point at which the file can have grown since the last run. Failures are
swallowed - housekeeping must never be the reason an update does not land.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Instances were started by hand and updated by hand, so they drifted behind
master silently. Now: PM2 supervises the dashboard and the clicker, a logon
task brings them up, and a 5-minute task pulls, rebuilds and restarts when
master moves.
Restarting on every push is only safe because the scheduler now survives it.
It was pure in-memory state (_global.__autoTrader), so any restart silently
stopped automated trading with the dashboard simply showing it as off. It now
mirrors running/action/symbol/stopAfterAll to the settings table, and
resumeSchedulerIfPersisted() picks it back up from the getClients() bootstrap.
No sync-wait was needed there: tick() already skips while a client reports
!syncComplete and while any account holds a position.
A failed build is never deployed — the build runs before anything restarts, so
a broken push leaves the previous build serving.
start-all and update-check both warm the app with a request afterwards. That is
load-bearing: getClients() is lazily bootstrapped, so until something makes an
HTTP request the Tradovate clients, the reporter and the resumed schedule never
start. That was already true of manual restarts.
Logic lives in Node so a macOS or Linux port only needs an equivalent of
install-autostart.ps1. Python deps are hash-guarded, so the common path is one
hash and one import with no network, and failure is non-fatal since only the
clicker needs them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
