Brandon LiandClaude Opus 5
54221bbc0c
Builds the pipeline the autobuyer needs: see the page, find an element, click it. extension/ — MV3 Chromium extension. Polls /api/autobuyer/status and, while on, scrapes the target tab's HTML and posts it back. Also serves locate requests: focuses the window, scrolls the element into view, and reports its position. host_permissions is scoped to tradeify plus localhost so it cannot read other sites — an empty target pattern would otherwise capture whatever tab happened to be active, including banking or mail. app/api/autobuyer/ — status toggle, capture store, and the locate request queue. CORS is open because the extension's origin changes every time an unpacked extension is reloaded. app/autobuyer/page.tsx — ON switch, source view (default) and a rendered view. The render uses sandbox="allow-scripts" without allow-same-origin: the page's own JS is needed because sites ship content at opacity:0 and fade it in, but the frame must not reach the dashboard's same-origin API routes, which serve firm credentials. clicker/ — Python CLI. Asks the extension where a selector is, adds the element rect to the window's screen position and the browser chrome height to get desktop coordinates, then clicks with a human motion model (curved path, eased velocity, occasional overshoot, dwell before press). Raises the browser application first, since macOS consumes a click on an unfocused window rather than delivering it. Refuses to click when the element is covered by an overlay, when the coordinates fall off-screen, or when the browser cannot be confirmed frontmost. Verified: API round-trips, capture pruning, locate claim-once semantics, motion geometry and timing, and focus activation — the last two against stubs, since pyautogui and pyobjc are not installed here. NOT verified end to end: Chrome is still running a stale build of the extension, so a locate request has never completed against a real page and no real click has been sent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
127 lines
5.8 KiB
Markdown
127 lines
5.8 KiB
Markdown
# Clicker
|
||
|
||
Moves the real desktop mouse and clicks an element you name with a CSS selector.
|
||
|
||
The extension can see the DOM but can't move the mouse; this script can move the
|
||
mouse but can't see inside Chrome. They meet at the dashboard API:
|
||
|
||
```
|
||
clicker --POST /api/autobuyer/locate------> "where is button.buy?"
|
||
extension --POST /api/autobuyer/locate/claim takes the request
|
||
extension raises the Chrome window, brings
|
||
the tab to the front, scrolls the
|
||
element to centre, measures it
|
||
extension --POST /api/autobuyer/locate/result desktop x,y
|
||
clicker --GET /api/autobuyer/locate?id=---> reads the answer
|
||
clicker moves the mouse, clicks
|
||
```
|
||
|
||
## Setup
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
On **macOS** you must grant Accessibility permission to whatever runs the script
|
||
(Terminal, iTerm, VS Code) — System Settings → Privacy & Security → Accessibility.
|
||
Without it `pyautogui` moves nothing and fails silently.
|
||
|
||
## Use
|
||
|
||
The AutoBuyer switch on the dashboard is the master arm: the script refuses to run
|
||
while it's off.
|
||
|
||
```bash
|
||
# Measure only — no mouse movement. Start here.
|
||
python clicker.py locate "button.buy"
|
||
|
||
# Move the cursor to the target but don't press.
|
||
python clicker.py click "button.buy" --dry-run
|
||
|
||
# Actually click.
|
||
python clicker.py click "button.buy"
|
||
|
||
# Pin it to a specific tab, and pick the 3rd match.
|
||
python clicker.py click ".trade-btn" --index 2 --url "https://tradeify.co/*"
|
||
```
|
||
|
||
| Flag | Meaning |
|
||
|---|---|
|
||
| `--url` | Chrome match pattern for the tab. Without it, the active tab is used. |
|
||
| `--index` | Which match, when the selector hits several (default 0). |
|
||
| `--api` | Dashboard URL (default `http://localhost:3000`). |
|
||
| `--timeout` | Seconds to wait for the extension (default 20). |
|
||
| `--scale` | CSS-to-desktop pixel ratio. Auto-detected; override if clicks land off. |
|
||
| `--dry-run` | Move the cursor, don't press. |
|
||
| `--force` | Click even when something is covering the element. |
|
||
| `--robotic` | Straight-line move and instant click, skipping the motion model. |
|
||
| `--seed` | Seed the motion RNG so a run replays identically (debugging). |
|
||
| `--no-activate` | Don't raise the browser first. The click may then be swallowed. |
|
||
|
||
## Window focus
|
||
|
||
A click on a window that isn't focused is consumed by the window manager
|
||
*activating* that window — it never reaches the control underneath. That's why an
|
||
automated click against a background Chrome appears to do nothing the first time
|
||
and work the second: the first click only brought Chrome forward.
|
||
|
||
The extension calls `chrome.windows.update({focused: true})`, but that only orders
|
||
windows **within** Chrome. If the frontmost *application* is your terminal — which
|
||
it is, since that's where you launched this — Chrome is still in the background.
|
||
|
||
So `focus.py` raises the browser application itself immediately before the press,
|
||
then confirms it actually came forward before committing to the click. If it can't
|
||
verify, it refuses rather than firing a click that would be eaten (exit code 3).
|
||
|
||
On macOS this uses `NSWorkspace` via pyobjc, which needs no Automation permission —
|
||
activating an app is not scripting it. Without pyobjc it falls back to `osascript`,
|
||
which does prompt for Automation permission the first time.
|
||
|
||
## Cursor motion
|
||
|
||
`humanize.py` moves the pointer the way a hand does rather than teleporting:
|
||
|
||
- a curved (cubic Bézier) path instead of a straight line, bowing to one side
|
||
- eased velocity — accelerate out, coast, decelerate in
|
||
- sub-pixel tremor that decays near the target, so the landing stays exact
|
||
- long throws (>260px) sometimes overshoot slightly and pull back
|
||
- a 60–170ms dwell after arriving, before the press
|
||
- the button held down 55–120ms rather than an instant down/up
|
||
|
||
This is about reliability as much as appearance. Plenty of web controls only arm
|
||
once they have actually been hovered — dropdowns, tooltip-gated buttons, custom
|
||
widgets — and a cursor that arrives and presses in the same tick can outrun the
|
||
page's own `mousemove` handlers. The dwell is what lets those catch up.
|
||
|
||
Note `pyautogui.PAUSE` is set to 0 on import: it otherwise sleeps 0.1s after
|
||
*every* call, which would add tens of seconds across a stepped path.
|
||
|
||
Exit codes: `0` ok, `1` error, `2` capture switch off, `3` refused to click
|
||
(covered element, or coordinates off-screen).
|
||
|
||
## Safety
|
||
|
||
- **Failsafe**: slam the cursor into a screen corner to abort mid-run.
|
||
- **Covered elements**: before reporting, the extension checks
|
||
`document.elementFromPoint()` at the target centre. If a cookie banner or modal
|
||
is on top, the click is refused rather than sent into the overlay — `--force`
|
||
overrides.
|
||
- **Off-screen check**: coordinates outside the display bounds are refused, which
|
||
catches a Chrome window on a second monitor or partly off the edge.
|
||
- The element is scrolled to the centre of the viewport before measuring, so a
|
||
target below the fold is handled rather than mis-clicked.
|
||
|
||
## Known limits
|
||
|
||
- **Latency** is bounded by the extension's poll interval (default 3s), so a click
|
||
takes a few seconds to fire. Drop the interval in the extension popup if that
|
||
matters.
|
||
- **Multi-monitor**: coordinates come from `window.screenX/screenY`, which are
|
||
relative to the primary display's origin. A Chrome window on a secondary monitor
|
||
usually still works, but verify with `locate` before trusting `click`.
|
||
- **Display scaling**: the scale factor is inferred by comparing the screen width
|
||
the OS reports against the one the browser reports. On macOS and unscaled Windows
|
||
this is 1:1. If clicks land at a consistent offset, set `--scale` explicitly.
|
||
- The measurement and the click are separate moments. If the page moves the element
|
||
in between (a re-render, a late-loading banner), the click lands where it *was*.
|