autofirmer-expanded/README.md

# AutoFirmer

Automated trading dashboard for prop firm accounts via Tradovate and TopstepX
(ProjectX Gateway) for CME futures, and DXtrade for 24/7 crypto/CFD firms
(Tradeify247).

---

## Setup

### Windows

In **PowerShell**, from wherever you want the instance to live:

```powershell
irm https://git.juicerroom.com/senofy/autofirmer-expanded/raw/branch/master/setup-windows.bat -OutFile setup-windows.bat
```

```powershell
.\setup-windows.bat
```

That is the whole install. It clones into an `autofirmer` folder beside itself,
installs anything missing (Git, Node, Python) via winget, installs dependencies,
builds, asks for the master dashboard URL and an instance name, and offers to
set up auto-start and auto-update.

Re-running it updates an existing checkout rather than cloning again, so it
doubles as a repair tool.

Two things it deliberately does, worth knowing:

- **Node LTS, not latest.** `better-sqlite3` ships prebuilt binaries for LTS
  only. On a newer major it falls back to compiling with node-gyp and fails
  without Visual Studio Build Tools, so the script warns before that happens.
- **It looks for tools off `PATH`.** A tool installed in the same session it is
  needed — or installed without its "add to PATH" option — is invisible to a
  running shell, so the script checks the standard install directories too.

### Raspberry Pi 5 / Debian / Ubuntu

```bash
curl -fsSLO https://git.juicerroom.com/senofy/autofirmer-expanded/raw/branch/master/setup-linux.sh
```

```bash
bash setup-linux.sh
```

Same idea, apt instead of winget. It installs Node 22 from NodeSource (Pi OS
ships one too old for Next 16), `xdotool` and `scrot` for the clicker, and puts
the Python dependencies in a virtualenv — Pi OS Bookworm enforces PEP 668, so a
system-wide `pip install` fails outright.

**The clicker needs X11.** Pi OS on a Pi 5 defaults to Wayland, which does not
let one client synthesise input into another or read the focused window, so
neither `xdotool` nor pyautogui can work there. The dashboard is unaffected, but
for the AutoBuyer:

```bash
sudo raspi-config    # Advanced Options -> Wayland -> X11, then reboot
```

`python clicker/diagnose.py` reports the session type, `DISPLAY` and whether
`xdotool` is present — run it before trusting a new machine.

The dashboard runs on port **3000**: `http://localhost:3000`.

---

## First-time setup

On first run, the SQLite database (`autotrader.sqlite`) is created automatically in the project root. No migrations need to be run manually.

Firms and their account types live on the master dashboard, not here — every
machine pulls the same catalog so they cannot drift apart. This machine only
stores the platform login for each firm.

1. On the **master dashboard**, open **Firms**, add the firm, and give it one
   account type per prefix + account size combination (profit target,
   consistency %, min trading days, and so on)
2. Set **Dashboard URL** on this machine's **Settings** page so it can reach the
   master. The catalog is pulled at startup and then every 60 seconds
3. On the main page, each firm from the catalog appears. Firms without stored
   credentials are greyed out — enter the Tradovate username and password in the
   fields beside the firm name and click **Save**. A firm whose platform is
   **TopstepX** on the master asks for an API key instead (generated in the
   TopstepX platform under Settings → API); the username is optional to store
   but the ProjectX login needs it before the firm can connect, so add it when
   you have it. Password is unused on TopstepX. A **DXtrade** firm asks for
   its DXtrade username and password (the ones behind the key icon on the
   firm's dashboard, not the dashboard login); the server URL and login domain
   come from the master
4. Accounts populate once the firm connects. Each is matched to an account type
   by its name prefix *and* its size, so a new account shows as **Unassigned**
   until its opening funding transaction has been fetched. Use the **Account
   Type** dropdown to pin one by hand; a manual pick is never overwritten

---

## Instruments: what each firm trades

Each firm's **Instruments** list on the master's Firms page is the one place
that says what it trades — there are no per-machine toggles or banned symbols
any more (the instance's Settings page shows the lists read-only).

- **Futures firms** (Tradovate, TopstepX) list CME roots — the same names on
  both platforms; each client turns a root into its own front-month contract.
- **DXtrade firms** list their own symbols (`ETH/USD`), whatever root a batch
  drew.

With the scheduler on **Auto**, each futures batch draws one instrument from
the batch firm's list, at random but weighted:

- **Efficiency** — one contract's typical daily move ÷ one tick of spread
  (from Yahoo's hourly candles over the past week, measured once a day).
  Commissions don't count — they barely differ between instruments.
- **Session timing** — at the open every instrument gets a 10% floor and the
  weights follow efficiency; toward the last entry (14:55 CT) the floor fades
  and the weight moves sharply to the fastest movers.
- **12-hour cap** — sizing is unchanged (a contract per $1,000 of leg, up to
  the type's max contracts); an instrument whose expected hold for the
  batch's leg is 12 h or more is out of the draw (e.g. 6E at a $1,000 leg,
  about a week).
- Micros stay out of the draw; drain and extra-day legs always trade MNQ.

Every draw logs its odds. `GET /api/instruments` shows each firm's list with
spread, daily move, hold and chance (`?simulate=10000` runs the real draw
with no orders), and the master's Firms page shows the same next to each
instrument.

Against a master from before the lists, an instance falls back to the old
rules: this machine's enabled instruments less each firm's banned symbols,
and a DXtrade firm's symbol map.

---

## DXtrade firms (Tradeify247)

A DXtrade firm is set up on the master's **Firms** page like any other, plus:

- **Server URL** (`https://dx.tradeify247.co/dxsca-web`) and **login domain**
  (usually `default`). Changing the URL re-keys the firm's accounts.
These settings only exist for DXtrade firms; futures firms keep their measured
per-contract fees and the CME calendar.

- **Commission %** of notional per side (Tradeify: `0.04`). The engine prices
  every fee off the fill with it.
- **Trades every day** — weekends and CME holidays included. Such a firm runs
  in its own scheduler *lane*: its own trading hours (closed only around the
  daily roll), its own wait-until-flat gate and its own batches, so a
  multi-hour crypto position never holds up the futures open.
- **Instruments** — the firm's own symbols (see above). The cheapest
  tradable one right now wins: spread plus both commissions, per unit of its
  volatility (hourly closes over the past week, measured once a day).
- **Target hold hours** — each leg's notional is sized from that volatility so
  its expected hold is about this long (blank = 12 h). The position never
  exceeds what the account's available funds can margin at the broker's
  rate for the symbol (Tradeify: 5:1 crypto, 2:1 stocks — read from DXtrade,
  nothing to set). A batch sizes its leg on every candidate and trades the
  cheapest (spread, commissions, swap) that reaches the target hold; if the
  leverage caps keep all of them short of it, the quickest.
- **Financing %/day** if the firm charges swap (Tradeify 1-Step: `0.033`).
  Each take-profit then also carries the swap a winning leg is expected to pay
  before it fills, so a win still clears the full leg.

Per account type, in the **Hard Daily Loss** column: the **daily loss limit**
(Tradeify: 3% of size — every leg then carries a stop that keeps the firm's
day inside it, less fees and a reserve of 10% of the limit or 0.1% of the
position, whichever is more) and the **static max drawdown** (Tradeify
1-Step: $6,000 on 100K). When hitting the day's loss limit would leave the
account within 15% of that loss above its max-drawdown floor, the leg is a
*last stand*: no daily-loss stop, running to its take-profit or the firm's
breach.

Before the scheduler trades a DXtrade firm, run the smoke test against a real
login (with this instance's client for the firm disconnected):

```bash
DX_BASE=https://dx.tradeify247.co/dxsca-web DX_USER=… DX_PASS=… DX_SYMBOLS=BTC/USD,ETH/USD node scripts/dxtrade-smoke.mjs
```

It is read-only unless `DX_CONFIRM=yes DX_ACCOUNT=<code>` are set too, which
adds one minimum-size live round trip.

`npm run check` runs the pure-logic checks (fees, stops, sizing, trading
hours, the DXtrade selector, the futures draw and instrument lists), plus local
HTTP/WebSocket checks against a temporary database.

### Outbound user agent

On first server load, each instance randomly selects a desktop Chrome,
Firefox, or Edge user agent and saves it as `user_agent` in its settings.
Settings → Network shows the selected value. It stays the same across requests,
WebSocket reconnects, page reloads, and server restarts, and travels with backups.
An existing saved value is reused; an absent or blank value is generated automatically.

All server Axios calls use `lib/http-client.ts`; import Axios from
`@/lib/http-client` when adding requests. Its defaults and interceptor also apply
to `axios.create()` clients, so per-request headers cannot accidentally replace
the saved user agent. ESLint rejects direct application imports of the Axios package.
Use `brokerWebSocket` or `signalrWebSocket` from `lib/outbound-proxy.ts` for broker
sockets, and `directWebSocket` for control sockets. All share the same user agent;
the broker proxy still applies only to broker traffic.

---

## The master dashboard

Set **Instance Name** and **Dashboard URL** on the Settings page and this
instance does two things with them: it reports its firm and account counts every
30 seconds, and it holds a WebSocket open to `<url>/ws/instance` so the master
can drive it.

Every report also carries this checkout's version — the HEAD commit and its
time, since updates arrive as `git pull` and there is no release number to
bump. The master shows it in a **Version** column and flags any machine running
something older than the newest build in the fleet, which is how you check at a
glance that every machine took the last update. An instance showing *old build*
predates version reporting entirely.

What the master can trigger here:

| | |
|---|---|
| Trade | Buy / Sell / Auto on a symbol, with the same "stop after all" option as the button on this dashboard |
| Stop | Halts the auto-trade scheduler |
| Run an automation | Queues a run for `clicker/runner.py`, exactly as the AutoBuyer page does |

Commands land in [`lib/master-link.ts`](lib/master-link.ts), which calls the same
functions the pages call — `runTrade`, `startScheduler`, `queueRun`. Nothing
reaches the broker by a path the local UI does not also use, and every command
is acked with its result or its error so the master shows what actually happened
rather than assuming.

The link is entirely optional. With no Dashboard URL configured it never starts,
and if the master is unreachable the instance keeps trading on its own and falls
back to the plain HTTP report until the socket comes back (retrying with backoff,
capped at a minute).

**Nothing authenticates this link in either direction.** Whatever answers at the
Dashboard URL can place orders on every account this instance holds, so point it
only at a master you control, and prefer `https` so the address cannot be
redirected in transit. If the socket will not stay up, the `connected` light on
the master is the only symptom — check this instance's logs for `[master-link]`.

## AutoBuyer (browser automation)

The AutoBuyer page drives a real browser to buy and reset prop-firm accounts. It
is three pieces, and all three must be running:

| Piece | What it does |
|---|---|
| the dashboard | Defines automations, queues runs, shows progress |
| `extension/` | A Chromium extension that reads the broker page and measures elements |
| `clicker/runner.py` | A desktop process that moves the real mouse and keyboard |

Automations are declared in `lib/automations.ts` — one entry per firm, with the
steps inside. Adding a button means editing that file; the page and the runner
pick it up from the server.

### 1. Install the extension

The setup scripts do this as their last step: give them the Chrome Enterprise
Core enrolment token from the Admin console (Devices → Chrome → Managed
browsers) and they enrol this machine's Chrome. The extension then arrives from
the console already set to force install, and Chrome keeps it updated. Run on
its own at any time:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\enroll-chrome-management.ps1
```

```bash
sudo scripts/install-extension-policy.sh
```

The enrolment token is built into both scripts — a token is reusable across any
number of machines — so there is nothing to type. Pass one explicitly to use a
different org. Chrome itself is installed first if the machine hasn't got it,
from Google's own enterprise MSI on Windows and the official .deb on Linux;
Chromium is not accepted as a substitute, because it has no cloud-management
support and can never be enrolled.

**A Raspberry Pi needs none of it.** Chrome is not published for ARM, but
Chromium is, and on Linux Chromium honours the same forcelist policy — the CRX
is pure JavaScript, so the architecture is irrelevant. `setup-linux.sh` writes
the policy and Chromium installs the extension from the update server on its
next start. Loading `extension/` unpacked still works and still self-reloads,
but it is no longer the only option there.

**Enrolment is a Windows problem only.** Windows refuses to force-install an
off-store extension unless the machine is domain-joined, Entra-joined or
enrolled in Chrome Enterprise Core — it reads the policy and blanks it — and
macOS wants MDM. Linux has no such requirement: a policy file under `/etc` is
enough, which is why a Raspberry Pi running Chromium installs this with no
Google account involved at all. On Linux, run `install-extension-policy.sh` and
skip enrolment entirely. See [deploy/README.md](deploy/README.md). Either way the
extension notices when
the dashboard is running newer code than it is: a policy install asks Chrome to
fetch the new CRX, an unpacked one reloads itself from disk. Clicking the reload
icon on its card still works and is the thing to try when something looks
stuck.

Adding a new firm also means adding its host to `host_permissions` in
`extension/manifest.json` and reloading. Without it the extension cannot read
that site, and every step fails to locate.

### 2. The clicker

`setup-windows.bat` installs its Python dependencies, and re-applies them
whenever `clicker/requirements.txt` changes. To do it by hand:

```powershell
python -m pip install -r clicker\requirements.txt
```

See `clicker/README.md` for the per-platform notes — display scaling and
foreground lock both matter on Windows — and for the verification sequence to
run before letting it click anything on a new machine.

### 3. Start the runner

Auto-start runs it under PM2, so normally there is nothing to do. To run it by
hand instead:

```powershell
python clicker\runner.py
```

It reports in every two seconds, and the dashboard greys out
the automation buttons when it is not there. A running Python process does not
reload when the source changes, so restart it after pulling updates; the
dashboard warns when its version is behind.

### 4. Captchas

A run that hits a captcha solves it through the **2captcha API**, not by clicking
2captcha's own browser extension. The API key is hardcoded in
`app/api/autobuyer/captcha/route.ts` — to bill a different account, set
`TWOCAPTCHA_API_KEY`, or PATCH `twocaptcha_api_key` to `/api/settings`. Either way
the key stays on the dashboard: the runner asks this machine to solve and never
handles it.

A `solveCaptcha` step then does three things, with no mouse involved: the
extension reads the widget's sitekey out of the page, the dashboard buys a token
from 2captcha, and the extension puts that token into the page's response field
and fires the page's own callback. reCAPTCHA v2 (including Enterprise), hCaptcha
and Turnstile are detected automatically.

If the 2captcha *extension* is still installed, disable it or turn off its
auto-solve. Two solvers racing on one widget is a good way to burn balance and
end up with a token the page has already replaced.

Solving costs money per captcha, and a failed solve is still billed. `timeoutSeconds`
on the step caps how long a run waits; 2captcha usually answers in 10–30s.

**The solver proxy.** Solves go through 2captcha's residential proxy, so the
challenge is answered from a residential address rather than a datacentre one.
The credentials are hardcoded in `app/api/autobuyer/captcha/route.ts`; override
them without touching code via `TWOCAPTCHA_PROXY`, or by PATCHing
`twocaptcha_proxy` to `/api/settings`. Either accepts `none` to solve proxyless.
Neither has a field on the Settings page — both are hardcoded, so there is
nothing to fill in. The run log names the address that
solved each captcha — the first thing to check when a token comes back and the
site rejects it anyway.

Two things that proxy does *not* do. It is not the browser's proxy: the token is
solved from the proxy's address but the form is submitted from this machine's,
and nothing makes those match. And the pool is not geo-pinned — despite the
`na.` hostname, test requests exited from Russia and Japan. For reCAPTCHA v2 and
Turnstile that mismatch is usually tolerated; if a site starts rejecting tokens
that 2captcha reports as solved, set the proxy to `none` and compare.

### Running it

Press an automation's button. Progress appears per step, and **Stop** halts a run
between steps.

There is no switch to throw first. Page capture follows whichever firm tab you
are looking at and stops when you switch away, and a run starts because someone
asked for one — from this page or from the master dashboard. The only thing that
gates a run is whether `clicker/runner.py` is up to claim it, which the page
shows as **Autoclicker daemon**.

That does mean a queued run executes as soon as the runner sees it, with no
second confirmation. Stop the runner if you want the machine idle.

The browser must be visible and frontmost while a run is in flight — the clicks
are real OS-level input, so the machine cannot be used for anything else, and a
dialog stealing focus fails the step. That makes an RDP session a poor host:
disconnecting can suspend the desktop and break clicks in ways that are hard to
diagnose.

## Keeping it running

`setup-windows.bat` offers to set this up for you at step 6. To enable it later,
or after declining:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\install-autostart.ps1
```

That puts the dashboard, the clicker and an update checker under PM2, adds a
Startup-folder entry so PM2 comes back at logon, and lets PM2's cron restart run
the update check every 5 minutes. It needs no administrator rights, changes
nothing machine-wide, and is safe to re-run.

```powershell
pm2 list                 # what is running
pm2 logs autofirmer      # dashboard output
pm2 logs clicker         # runner output
```

**Why not a Windows service.** The clicker sends real mouse and keyboard input
and has to own a desktop. A service runs in session 0, which has none, so the
clicks would go nowhere. Everything therefore runs in your logged-in session —
which also means an unattended reboot leaves the instance down until somebody
logs in.

To stop it starting at logon, delete the `AutoFirmer.cmd` shortcut from your
Startup folder (`shell:startup` in the Run dialog).

To start everything by hand without waiting for a logon:

```powershell
node scripts\start-all.mjs
```

`start-autofirmer.bat` still runs the dashboard in a visible window without PM2,
which is the easier thing to watch when a build is misbehaving.


## Firewall

To restrict port 3000 to a specific trusted IP only:

```powershell
New-NetFirewallRule -DisplayName "AutoFirmer" -Direction Inbound -Protocol TCP -LocalPort 3000 -RemoteAddress <your-ip> -Action Allow
```

Or open it to all inbound (less secure):

```powershell
New-NetFirewallRule -DisplayName "AutoFirmer" -Direction Inbound -Protocol TCP -LocalPort 3000 -Action Allow
```

---

## Publishing extension updates

The extension is signed and served from the droplet, which is what lets Chrome
update it on its own. Everything else — the dashboard, the clicker — stays on
the trading machine.

```bash
# on the droplet, in /opt/autofirmer
git pull && ./deploy/deploy.sh
```

That packs `extension/` at the version in its `manifest.json` — **bump it or
nothing updates** — signs it, and publishes the CRX and update manifest. See
[deploy/README.md](deploy/README.md).

---

## Updating

Once auto-start is installed, nothing here is manual. Every minute (the
installer's default interval) the update check fetches `master`. When there is something to do, it starts a background
worker. The worker pulls, reinstalls dependencies if `package-lock.json` or
`clicker/requirements.txt` changed, rebuilds, restarts every instance, and
restarts the clicker if anything under `clicker/` changed.

**Why a check and a separate worker.** PM2 runs the check as a cron *restart*,
and a restart kills a run that is still going at the next minute boundary. An
npm install or a build takes longer than that, so doing the work in the cron
process itself got it killed half-way through. The check finishes in seconds,
and the worker runs outside PM2's process tree, where the next tick can't
reach it.

**Deployed means built, not pulled.** `scripts/.built-commit` records the commit
`.next` was last built from, and the check compares `master` with that, not with
HEAD. A pull that was never built is therefore finished on the next check, and
so is a missing `.next/BUILD_ID`. The first time this version runs on a machine
it judges the build already on disk: one that is complete, newer than HEAD and
already serving is adopted, and anything else is rebuilt.

**A failed build is never deployed.** The build runs before anything restarts,
and the previous build is set aside as `.next-prev` first. A failed build is
rolled back to it, so the box is left serving the previous build and a later
restart or reboot still has one. A failure is retried 15 minutes later, then
hourly until it succeeds or a new commit lands. `scripts/.update-failed` records
which step failed and when the next try is due.

### Checking it is alive

`scripts/.last-check` is rewritten on every check, whether or not anything came
down:

```
2026-08-31T00:43:27.155Z  up to date at 27a61fd2
```

This exists because the alternatives mislead. `update.log` only records real
events, so it stays empty for days when nothing is pushed — and PM2 reports a
cron-restart process as `stopped` with a restart count of `0` even while it is
firing on schedule. Neither is evidence of a problem; `.last-check` is the
signal to trust.

While a worker is running, `.last-check` says `update in progress` with its pid.
After a failure it says `waiting to retry` with the time of the next attempt.

Everything that actually happens is appended to `scripts/update.log`. The full
npm and build output of the latest worker run is in `scripts/update-worker.log`,
which is the first place to look when an update failed. To see what it would do
without touching anything:

```powershell
node scripts\update-check.mjs --dry-run
```

To apply an update immediately in the foreground, bypassing the retry wait:

```powershell
node scripts\update-check.mjs --apply
```

Without `--apply` it does what the schedule does: it checks, starts a worker if
there is anything to do, and returns straight away.

### What reloads itself, and what does not

- **The extension** updates itself now. `/api/autobuyer/status` reports the
  version the dashboard's own checkout ships, and when the running build is
  older the extension acts on it: a policy install asks Chrome to fetch the new
  CRX from the update server, an unpacked one calls `chrome.runtime.reload()`
  and comes back on the files the update check pulled. The unpacked path can
  only run code already on disk, so if it reloads before the pull lands it says
  so in the popup and waits rather than restarting in a loop.
- **The scheduler is fine now.** It persists to the settings table and resumes
  after a restart, so an update no longer silently stops automated trading.
- **The runner** does not. It is a long-lived Python process; the update check
  restarts it when anything under `clicker/` changes, and the dashboard warns
  when the version it reports is behind what the server expects.

The dashboard reports the version it sees from the extension and the runner, and
warns when either is behind. Most AutoBuyer bugs that look mysterious are the
extension still running the previous code.