Updater heals itself: PM2 cron revived, Update now, failure reasons

PM2's cron for autofirmer-update dies inside a live daemon (six Windows
machines in October), and nothing on the machine noticed. The app now
watches scripts/.last-check and, past 10 minutes, runs
update-check.mjs --heal: a checker PM2 still lists is restarted (which
re-registers its cron), one it lost is registered again, and one whose
errored entry still holds a live pid is left alone, since PM2 would kill
that pid. The heal scrubs PM2's per-app fields from its environment first.

The master's new 'update-now' command runs --heal --now, skipping a
failed update's retry wait. A failed pull, npm install or build records
one line of reason (or the chown to run when a file isn't ours), which
rides on .last-check to the master. A package-lock.json rewritten by npm
is put back before the pull. The fetch timeout drops under the cron's
minute. scripts/.no-auto-update opts a checkout out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Li
2026-10-19 00:41:04 -05:00
co-authored by Claude Opus 5.5
parent c0899620d4
commit 13e03c3d0f
6 changed files with 396 additions and 23 deletions
+4
View File
@@ -12,6 +12,7 @@ import { startReporter, sendReport } from './reporter';
import { startBackupSync } from './backup-sync';
import { applyBrokerHttpDefaults, applyBrokerProxy } from './outbound-proxy';
import { startMasterLink } from './master-link';
import { startUpdaterWatchdog } from './updater-watchdog';
import { resumeSchedulerIfPersisted } from './auto-trade';
import { superviseQuotes, SUPERVISE_INTERVAL_MS, type QuoteClient } from './live-quotes';
@@ -227,6 +228,9 @@ export function getClients(): Map<string, PlatformClient> {
// Start master dashboard reporter
startReporter();
// Bring this machine's auto-updater back if its PM2 cron stops firing.
startUpdaterWatchdog();
// Push settings+credentials backups to the master, so a dead VPS
// can be replaced and this identity adopted.
startBackupSync();