Add autobuyer page capture, browser extension, and desktop clicker

Builds the pipeline the autobuyer needs: see the page, find an element,
click it.

extension/ — MV3 Chromium extension. Polls /api/autobuyer/status and,
while on, scrapes the target tab's HTML and posts it back. Also serves
locate requests: focuses the window, scrolls the element into view, and
reports its position. host_permissions is scoped to tradeify plus
localhost so it cannot read other sites — an empty target pattern would
otherwise capture whatever tab happened to be active, including banking
or mail.

app/api/autobuyer/ — status toggle, capture store, and the locate request
queue. CORS is open because the extension's origin changes every time an
unpacked extension is reloaded.

app/autobuyer/page.tsx — ON switch, source view (default) and a rendered
view. The render uses sandbox="allow-scripts" without allow-same-origin:
the page's own JS is needed because sites ship content at opacity:0 and
fade it in, but the frame must not reach the dashboard's same-origin API
routes, which serve firm credentials.

clicker/ — Python CLI. Asks the extension where a selector is, adds the
element rect to the window's screen position and the browser chrome
height to get desktop coordinates, then clicks with a human motion model
(curved path, eased velocity, occasional overshoot, dwell before press).
Raises the browser application first, since macOS consumes a click on an
unfocused window rather than delivering it.

Refuses to click when the element is covered by an overlay, when the
coordinates fall off-screen, or when the browser cannot be confirmed
frontmost.

Verified: API round-trips, capture pruning, locate claim-once semantics,
motion geometry and timing, and focus activation — the last two against
stubs, since pyautogui and pyobjc are not installed here. NOT verified
end to end: Chrome is still running a stale build of the extension, so a
locate request has never completed against a real page and no real click
has been sent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Brandon Li
2026-08-27 16:15:18 -05:00
co-authored by Claude Opus 5
parent 24a54c6642
commit 54221bbc0c
19 changed files with 1752 additions and 2 deletions
+126
View File
@@ -0,0 +1,126 @@
# Clicker
Moves the real desktop mouse and clicks an element you name with a CSS selector.
The extension can see the DOM but can't move the mouse; this script can move the
mouse but can't see inside Chrome. They meet at the dashboard API:
```
clicker --POST /api/autobuyer/locate------> "where is button.buy?"
extension --POST /api/autobuyer/locate/claim takes the request
extension raises the Chrome window, brings
the tab to the front, scrolls the
element to centre, measures it
extension --POST /api/autobuyer/locate/result desktop x,y
clicker --GET /api/autobuyer/locate?id=---> reads the answer
clicker moves the mouse, clicks
```
## Setup
```bash
pip install -r requirements.txt
```
On **macOS** you must grant Accessibility permission to whatever runs the script
(Terminal, iTerm, VS Code) — System Settings → Privacy & Security → Accessibility.
Without it `pyautogui` moves nothing and fails silently.
## Use
The AutoBuyer switch on the dashboard is the master arm: the script refuses to run
while it's off.
```bash
# Measure only — no mouse movement. Start here.
python clicker.py locate "button.buy"
# Move the cursor to the target but don't press.
python clicker.py click "button.buy" --dry-run
# Actually click.
python clicker.py click "button.buy"
# Pin it to a specific tab, and pick the 3rd match.
python clicker.py click ".trade-btn" --index 2 --url "https://tradeify.co/*"
```
| Flag | Meaning |
|---|---|
| `--url` | Chrome match pattern for the tab. Without it, the active tab is used. |
| `--index` | Which match, when the selector hits several (default 0). |
| `--api` | Dashboard URL (default `http://localhost:3000`). |
| `--timeout` | Seconds to wait for the extension (default 20). |
| `--scale` | CSS-to-desktop pixel ratio. Auto-detected; override if clicks land off. |
| `--dry-run` | Move the cursor, don't press. |
| `--force` | Click even when something is covering the element. |
| `--robotic` | Straight-line move and instant click, skipping the motion model. |
| `--seed` | Seed the motion RNG so a run replays identically (debugging). |
| `--no-activate` | Don't raise the browser first. The click may then be swallowed. |
## Window focus
A click on a window that isn't focused is consumed by the window manager
*activating* that window — it never reaches the control underneath. That's why an
automated click against a background Chrome appears to do nothing the first time
and work the second: the first click only brought Chrome forward.
The extension calls `chrome.windows.update({focused: true})`, but that only orders
windows **within** Chrome. If the frontmost *application* is your terminal — which
it is, since that's where you launched this — Chrome is still in the background.
So `focus.py` raises the browser application itself immediately before the press,
then confirms it actually came forward before committing to the click. If it can't
verify, it refuses rather than firing a click that would be eaten (exit code 3).
On macOS this uses `NSWorkspace` via pyobjc, which needs no Automation permission —
activating an app is not scripting it. Without pyobjc it falls back to `osascript`,
which does prompt for Automation permission the first time.
## Cursor motion
`humanize.py` moves the pointer the way a hand does rather than teleporting:
- a curved (cubic Bézier) path instead of a straight line, bowing to one side
- eased velocity — accelerate out, coast, decelerate in
- sub-pixel tremor that decays near the target, so the landing stays exact
- long throws (>260px) sometimes overshoot slightly and pull back
- a 60170ms dwell after arriving, before the press
- the button held down 55120ms rather than an instant down/up
This is about reliability as much as appearance. Plenty of web controls only arm
once they have actually been hovered — dropdowns, tooltip-gated buttons, custom
widgets — and a cursor that arrives and presses in the same tick can outrun the
page's own `mousemove` handlers. The dwell is what lets those catch up.
Note `pyautogui.PAUSE` is set to 0 on import: it otherwise sleeps 0.1s after
*every* call, which would add tens of seconds across a stepped path.
Exit codes: `0` ok, `1` error, `2` capture switch off, `3` refused to click
(covered element, or coordinates off-screen).
## Safety
- **Failsafe**: slam the cursor into a screen corner to abort mid-run.
- **Covered elements**: before reporting, the extension checks
`document.elementFromPoint()` at the target centre. If a cookie banner or modal
is on top, the click is refused rather than sent into the overlay — `--force`
overrides.
- **Off-screen check**: coordinates outside the display bounds are refused, which
catches a Chrome window on a second monitor or partly off the edge.
- The element is scrolled to the centre of the viewport before measuring, so a
target below the fold is handled rather than mis-clicked.
## Known limits
- **Latency** is bounded by the extension's poll interval (default 3s), so a click
takes a few seconds to fire. Drop the interval in the extension popup if that
matters.
- **Multi-monitor**: coordinates come from `window.screenX/screenY`, which are
relative to the primary display's origin. A Chrome window on a secondary monitor
usually still works, but verify with `locate` before trusting `click`.
- **Display scaling**: the scale factor is inferred by comparing the screen width
the OS reports against the one the browser reports. On macOS and unscaled Windows
this is 1:1. If clicks land at a consistent offset, set `--scale` explicitly.
- The measurement and the click are separate moments. If the page moves the element
in between (a re-render, a late-loading banner), the click lands where it *was*.