Add autobuyer page capture, browser extension, and desktop clicker
Builds the pipeline the autobuyer needs: see the page, find an element, click it. extension/ — MV3 Chromium extension. Polls /api/autobuyer/status and, while on, scrapes the target tab's HTML and posts it back. Also serves locate requests: focuses the window, scrolls the element into view, and reports its position. host_permissions is scoped to tradeify plus localhost so it cannot read other sites — an empty target pattern would otherwise capture whatever tab happened to be active, including banking or mail. app/api/autobuyer/ — status toggle, capture store, and the locate request queue. CORS is open because the extension's origin changes every time an unpacked extension is reloaded. app/autobuyer/page.tsx — ON switch, source view (default) and a rendered view. The render uses sandbox="allow-scripts" without allow-same-origin: the page's own JS is needed because sites ship content at opacity:0 and fade it in, but the frame must not reach the dashboard's same-origin API routes, which serve firm credentials. clicker/ — Python CLI. Asks the extension where a selector is, adds the element rect to the window's screen position and the browser chrome height to get desktop coordinates, then clicks with a human motion model (curved path, eased velocity, occasional overshoot, dwell before press). Raises the browser application first, since macOS consumes a click on an unfocused window rather than delivering it. Refuses to click when the element is covered by an overlay, when the coordinates fall off-screen, or when the browser cannot be confirmed frontmost. Verified: API round-trips, capture pruning, locate claim-once semantics, motion geometry and timing, and focus activation — the last two against stubs, since pyautogui and pyobjc are not installed here. NOT verified end to end: Chrome is still running a stale build of the extension, so a locate request has never completed against a real page and no real click has been sent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Brandon Li
co-authored by
Claude Opus 5
Claude Opus 5
parent
24a54c6642
commit
54221bbc0c
@@ -0,0 +1,126 @@
|
||||
# Clicker
|
||||
|
||||
Moves the real desktop mouse and clicks an element you name with a CSS selector.
|
||||
|
||||
The extension can see the DOM but can't move the mouse; this script can move the
|
||||
mouse but can't see inside Chrome. They meet at the dashboard API:
|
||||
|
||||
```
|
||||
clicker --POST /api/autobuyer/locate------> "where is button.buy?"
|
||||
extension --POST /api/autobuyer/locate/claim takes the request
|
||||
extension raises the Chrome window, brings
|
||||
the tab to the front, scrolls the
|
||||
element to centre, measures it
|
||||
extension --POST /api/autobuyer/locate/result desktop x,y
|
||||
clicker --GET /api/autobuyer/locate?id=---> reads the answer
|
||||
clicker moves the mouse, clicks
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
On **macOS** you must grant Accessibility permission to whatever runs the script
|
||||
(Terminal, iTerm, VS Code) — System Settings → Privacy & Security → Accessibility.
|
||||
Without it `pyautogui` moves nothing and fails silently.
|
||||
|
||||
## Use
|
||||
|
||||
The AutoBuyer switch on the dashboard is the master arm: the script refuses to run
|
||||
while it's off.
|
||||
|
||||
```bash
|
||||
# Measure only — no mouse movement. Start here.
|
||||
python clicker.py locate "button.buy"
|
||||
|
||||
# Move the cursor to the target but don't press.
|
||||
python clicker.py click "button.buy" --dry-run
|
||||
|
||||
# Actually click.
|
||||
python clicker.py click "button.buy"
|
||||
|
||||
# Pin it to a specific tab, and pick the 3rd match.
|
||||
python clicker.py click ".trade-btn" --index 2 --url "https://tradeify.co/*"
|
||||
```
|
||||
|
||||
| Flag | Meaning |
|
||||
|---|---|
|
||||
| `--url` | Chrome match pattern for the tab. Without it, the active tab is used. |
|
||||
| `--index` | Which match, when the selector hits several (default 0). |
|
||||
| `--api` | Dashboard URL (default `http://localhost:3000`). |
|
||||
| `--timeout` | Seconds to wait for the extension (default 20). |
|
||||
| `--scale` | CSS-to-desktop pixel ratio. Auto-detected; override if clicks land off. |
|
||||
| `--dry-run` | Move the cursor, don't press. |
|
||||
| `--force` | Click even when something is covering the element. |
|
||||
| `--robotic` | Straight-line move and instant click, skipping the motion model. |
|
||||
| `--seed` | Seed the motion RNG so a run replays identically (debugging). |
|
||||
| `--no-activate` | Don't raise the browser first. The click may then be swallowed. |
|
||||
|
||||
## Window focus
|
||||
|
||||
A click on a window that isn't focused is consumed by the window manager
|
||||
*activating* that window — it never reaches the control underneath. That's why an
|
||||
automated click against a background Chrome appears to do nothing the first time
|
||||
and work the second: the first click only brought Chrome forward.
|
||||
|
||||
The extension calls `chrome.windows.update({focused: true})`, but that only orders
|
||||
windows **within** Chrome. If the frontmost *application* is your terminal — which
|
||||
it is, since that's where you launched this — Chrome is still in the background.
|
||||
|
||||
So `focus.py` raises the browser application itself immediately before the press,
|
||||
then confirms it actually came forward before committing to the click. If it can't
|
||||
verify, it refuses rather than firing a click that would be eaten (exit code 3).
|
||||
|
||||
On macOS this uses `NSWorkspace` via pyobjc, which needs no Automation permission —
|
||||
activating an app is not scripting it. Without pyobjc it falls back to `osascript`,
|
||||
which does prompt for Automation permission the first time.
|
||||
|
||||
## Cursor motion
|
||||
|
||||
`humanize.py` moves the pointer the way a hand does rather than teleporting:
|
||||
|
||||
- a curved (cubic Bézier) path instead of a straight line, bowing to one side
|
||||
- eased velocity — accelerate out, coast, decelerate in
|
||||
- sub-pixel tremor that decays near the target, so the landing stays exact
|
||||
- long throws (>260px) sometimes overshoot slightly and pull back
|
||||
- a 60–170ms dwell after arriving, before the press
|
||||
- the button held down 55–120ms rather than an instant down/up
|
||||
|
||||
This is about reliability as much as appearance. Plenty of web controls only arm
|
||||
once they have actually been hovered — dropdowns, tooltip-gated buttons, custom
|
||||
widgets — and a cursor that arrives and presses in the same tick can outrun the
|
||||
page's own `mousemove` handlers. The dwell is what lets those catch up.
|
||||
|
||||
Note `pyautogui.PAUSE` is set to 0 on import: it otherwise sleeps 0.1s after
|
||||
*every* call, which would add tens of seconds across a stepped path.
|
||||
|
||||
Exit codes: `0` ok, `1` error, `2` capture switch off, `3` refused to click
|
||||
(covered element, or coordinates off-screen).
|
||||
|
||||
## Safety
|
||||
|
||||
- **Failsafe**: slam the cursor into a screen corner to abort mid-run.
|
||||
- **Covered elements**: before reporting, the extension checks
|
||||
`document.elementFromPoint()` at the target centre. If a cookie banner or modal
|
||||
is on top, the click is refused rather than sent into the overlay — `--force`
|
||||
overrides.
|
||||
- **Off-screen check**: coordinates outside the display bounds are refused, which
|
||||
catches a Chrome window on a second monitor or partly off the edge.
|
||||
- The element is scrolled to the centre of the viewport before measuring, so a
|
||||
target below the fold is handled rather than mis-clicked.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **Latency** is bounded by the extension's poll interval (default 3s), so a click
|
||||
takes a few seconds to fire. Drop the interval in the extension popup if that
|
||||
matters.
|
||||
- **Multi-monitor**: coordinates come from `window.screenX/screenY`, which are
|
||||
relative to the primary display's origin. A Chrome window on a secondary monitor
|
||||
usually still works, but verify with `locate` before trusting `click`.
|
||||
- **Display scaling**: the scale factor is inferred by comparing the screen width
|
||||
the OS reports against the one the browser reports. On macOS and unscaled Windows
|
||||
this is 1:1. If clicks land at a consistent offset, set `--scale` explicitly.
|
||||
- The measurement and the click are separate moments. If the page moves the element
|
||||
in between (a re-render, a late-loading banner), the click lands where it *was*.
|
||||
@@ -0,0 +1,223 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Desktop mouse control for the AutoFirmer autobuyer.
|
||||
|
||||
The browser extension can see the DOM but can't move the mouse; this script can
|
||||
move the mouse but can't see inside Chrome. They meet at the dashboard API:
|
||||
|
||||
clicker --POST /api/autobuyer/locate--> "where is button.buy?"
|
||||
extension raises the window, scrolls the
|
||||
element into view, measures it
|
||||
clicker --GET /api/autobuyer/locate--> desktop x,y
|
||||
clicker moves the real mouse and clicks
|
||||
|
||||
Usage:
|
||||
python clicker.py locate "button.buy" # measure only, no clicking
|
||||
python clicker.py click "button.buy" # measure, then really click
|
||||
python clicker.py click ".btn" --index 2 --url "https://tradeify.co/*"
|
||||
python clicker.py click "button.buy" --dry-run # move the cursor, don't press
|
||||
|
||||
Requires the AutoBuyer page switch to be ON — that's the master arming switch.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import focus
|
||||
|
||||
DEFAULT_API = "http://localhost:3000"
|
||||
POLL_INTERVAL = 0.25
|
||||
|
||||
|
||||
class DashboardError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class Dashboard:
|
||||
"""Thin client for the autobuyer endpoints."""
|
||||
|
||||
def __init__(self, base: str, timeout: float = 10.0):
|
||||
self.base = base.rstrip("/")
|
||||
self.timeout = timeout
|
||||
|
||||
def _request(self, path: str, method: str = "GET", payload: dict | None = None) -> dict:
|
||||
data = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {"Content-Type": "application/json"} if data else {}
|
||||
req = urllib.request.Request(self.base + path, data=data, headers=headers, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=self.timeout) as resp:
|
||||
return json.loads(resp.read().decode())
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = exc.read().decode(errors="replace")
|
||||
try:
|
||||
message = json.loads(body).get("error", body)
|
||||
except json.JSONDecodeError:
|
||||
message = body
|
||||
raise DashboardError(f"{method} {path} -> HTTP {exc.code}: {message}") from None
|
||||
except urllib.error.URLError as exc:
|
||||
raise DashboardError(f"Cannot reach {self.base} — {exc.reason}") from None
|
||||
|
||||
def status(self) -> dict:
|
||||
return self._request("/api/autobuyer/status")
|
||||
|
||||
def locate(self, selector: str, index: int, url_pattern: str, timeout: float) -> dict:
|
||||
"""Queue a lookup and block until the extension answers it."""
|
||||
queued = self._request(
|
||||
"/api/autobuyer/locate",
|
||||
"POST",
|
||||
{"selector": selector, "index": index, "urlPattern": url_pattern},
|
||||
)
|
||||
request_id = queued["id"]
|
||||
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
row = self._request(f"/api/autobuyer/locate?id={request_id}")
|
||||
if row["status"] == "done":
|
||||
return row["result"]
|
||||
if row["status"] == "error":
|
||||
raise DashboardError(f"Extension could not locate it: {row['error']}")
|
||||
time.sleep(POLL_INTERVAL)
|
||||
|
||||
raise DashboardError(
|
||||
f"No answer within {timeout:g}s. Is the extension installed, is Chrome "
|
||||
f"running, and is the AutoBuyer switch ON?"
|
||||
)
|
||||
|
||||
|
||||
def scale_factor(found: dict, override: float | None) -> float:
|
||||
"""CSS pixels and desktop pixels are the same on macOS and on unscaled Windows,
|
||||
but Windows display scaling and some Linux setups break that. Compare the
|
||||
screen size the browser reports against the one the OS reports."""
|
||||
if override is not None:
|
||||
return override
|
||||
|
||||
try:
|
||||
import pyautogui
|
||||
except ImportError:
|
||||
return 1.0 # `locate` is useful without the mouse library installed
|
||||
|
||||
os_width = pyautogui.size().width
|
||||
css_width = (found.get("screenSize") or {}).get("width")
|
||||
if not css_width:
|
||||
return 1.0
|
||||
ratio = os_width / css_width
|
||||
# Only trust a clean-ish ratio; anything odd means a multi-monitor layout we
|
||||
# shouldn't guess at, so fall back to 1:1 and let --scale override.
|
||||
return ratio if 0.4 < ratio < 4.0 else 1.0
|
||||
|
||||
|
||||
def describe(found: dict, factor: float) -> str:
|
||||
x, y = found["screen"]["x"] * factor, found["screen"]["y"] * factor
|
||||
lines = [
|
||||
f" matched <{found['tag']}> {found['text']!r}"
|
||||
+ (f" (1 of {found['matchCount']})" if found["matchCount"] > 1 else ""),
|
||||
f" page {found['url']}",
|
||||
f" viewport x={found['viewport']['x']:.0f} y={found['viewport']['y']:.0f}"
|
||||
f" {found['viewport']['width']:.0f}x{found['viewport']['height']:.0f}",
|
||||
f" desktop x={x:.0f} y={y:.0f}" + (f" (scale {factor:g})" if factor != 1 else ""),
|
||||
]
|
||||
if found.get("covered"):
|
||||
lines.append(f" WARNING something is on top of it: {found['coveredBy']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("action", choices=["locate", "click"],
|
||||
help="locate = measure only; click = measure then click")
|
||||
parser.add_argument("selector", help="CSS selector of the target element")
|
||||
parser.add_argument("--index", type=int, default=0, help="which match, if the selector hits several (default 0)")
|
||||
parser.add_argument("--url", default="", help='Chrome match pattern for the tab, e.g. "https://tradeify.co/*"')
|
||||
parser.add_argument("--api", default=DEFAULT_API, help=f"dashboard URL (default {DEFAULT_API})")
|
||||
parser.add_argument("--timeout", type=float, default=20.0, help="seconds to wait for the extension (default 20)")
|
||||
parser.add_argument("--scale", type=float, default=None, help="CSS-to-desktop pixel ratio (default: auto-detect)")
|
||||
parser.add_argument("--dry-run", action="store_true", help="move the cursor to the target but do not press")
|
||||
parser.add_argument("--force", action="store_true", help="click even if something is covering the element")
|
||||
parser.add_argument("--robotic", action="store_true",
|
||||
help="straight-line move and instant click, skipping the human motion model")
|
||||
parser.add_argument("--seed", type=int, default=None,
|
||||
help="seed the motion RNG so a run is reproducible (for debugging)")
|
||||
parser.add_argument("--no-activate", action="store_true",
|
||||
help="do not raise the browser first (the click may be eaten by window activation)")
|
||||
args = parser.parse_args()
|
||||
|
||||
dash = Dashboard(args.api)
|
||||
|
||||
try:
|
||||
if not dash.status().get("enabled"):
|
||||
print("AutoBuyer capture is OFF — turn it on from the dashboard first.", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
print(f"Locating {args.selector!r} …", flush=True)
|
||||
found = dash.locate(args.selector, args.index, args.url, args.timeout)
|
||||
except DashboardError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.action == "locate":
|
||||
print(describe(found, scale_factor(found, args.scale)))
|
||||
return 0
|
||||
|
||||
if found.get("covered") and not args.force:
|
||||
print(describe(found, 1.0))
|
||||
print("\nRefusing to click: the element is covered — the click would hit "
|
||||
f"{found['coveredBy']} instead. Pass --force to click anyway.", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
try:
|
||||
import pyautogui
|
||||
except ImportError:
|
||||
print("error: pyautogui is not installed — run: pip install -r requirements.txt", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# Slamming the cursor into a screen corner aborts the script.
|
||||
pyautogui.FAILSAFE = True
|
||||
|
||||
factor = scale_factor(found, args.scale)
|
||||
x = found["screen"]["x"] * factor
|
||||
y = found["screen"]["y"] * factor
|
||||
|
||||
screen_w, screen_h = pyautogui.size()
|
||||
if not (0 <= x < screen_w and 0 <= y < screen_h):
|
||||
print(describe(found, factor))
|
||||
print(f"\nerror: target ({x:.0f}, {y:.0f}) is off-screen ({screen_w}x{screen_h}). "
|
||||
f"Is the Chrome window partly off the display, or on a second monitor?", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
print(describe(found, factor))
|
||||
|
||||
# A click on a background window is eaten by the window manager activating it,
|
||||
# so the first attempt silently does nothing. Raise the browser first, and do it
|
||||
# here rather than before locating: the measurement takes seconds, and focus is
|
||||
# only required at the moment of the press.
|
||||
if not args.no_activate:
|
||||
focused = focus.ensure_frontmost()
|
||||
print(f" focus {focused.detail}")
|
||||
if not focused.ok:
|
||||
print("\nRefusing to click: the browser is not frontmost, so the click "
|
||||
"would be consumed activating its window instead of pressing the "
|
||||
"element. Pass --no-activate to override.", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
if args.robotic:
|
||||
pyautogui.moveTo(x, y, duration=0.25)
|
||||
if not args.dry_run:
|
||||
pyautogui.click()
|
||||
else:
|
||||
import humanize
|
||||
rng = random.Random(args.seed) if args.seed is not None else random.Random()
|
||||
humanize.click(x, y, rng=rng, press=not args.dry_run)
|
||||
|
||||
if args.dry_run:
|
||||
print("\ndry run — cursor moved, no click sent.")
|
||||
else:
|
||||
print("\nclicked.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Bring the browser to the front before clicking.
|
||||
|
||||
macOS (and Windows, to a lesser degree) treats a click on an unfocused window as
|
||||
an activation gesture: the click raises the window and is swallowed there, never
|
||||
reaching the control underneath. So an automated click against a background Chrome
|
||||
does nothing at all the first time, then works on the second attempt — which looks
|
||||
like a flaky clicker but is really the window manager doing its job.
|
||||
|
||||
The extension already calls chrome.windows.update({focused: true}), but that only
|
||||
orders windows *within* Chrome. If the frontmost application is your terminal —
|
||||
which it is, because that's where this script was launched — Chrome as a whole is
|
||||
still in the background. This module raises the application itself.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
# Chrome ships under several bundle ids; accept whichever is installed.
|
||||
MAC_BUNDLES = (
|
||||
"com.google.Chrome",
|
||||
"com.google.Chrome.beta",
|
||||
"com.google.Chrome.dev",
|
||||
"com.google.Chrome.canary",
|
||||
"com.brave.Browser",
|
||||
"com.microsoft.edgemac",
|
||||
)
|
||||
|
||||
SETTLE = 0.20 # let the window manager finish raising before measuring or clicking
|
||||
|
||||
|
||||
class FocusResult:
|
||||
def __init__(self, ok: bool, detail: str):
|
||||
self.ok = ok
|
||||
self.detail = detail
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return self.ok
|
||||
|
||||
|
||||
def _mac_workspace():
|
||||
"""NSWorkspace via pyobjc. Unlike AppleScript this needs no Automation
|
||||
permission — activating an app is not scripting it."""
|
||||
try:
|
||||
from AppKit import NSWorkspace
|
||||
except ImportError:
|
||||
return None
|
||||
return NSWorkspace.sharedWorkspace()
|
||||
|
||||
|
||||
def frontmost() -> str | None:
|
||||
"""Bundle id (macOS) or process name of the frontmost application."""
|
||||
if sys.platform == "darwin":
|
||||
ws = _mac_workspace()
|
||||
if ws is None:
|
||||
return None
|
||||
app = ws.frontmostApplication()
|
||||
return app.bundleIdentifier() if app else None
|
||||
return None
|
||||
|
||||
|
||||
def _mac_activate() -> FocusResult:
|
||||
ws = _mac_workspace()
|
||||
if ws is None:
|
||||
# pyobjc's AppKit isn't present. osascript works but may prompt for
|
||||
# Automation permission the first time.
|
||||
try:
|
||||
subprocess.run(
|
||||
["osascript", "-e", 'tell application "Google Chrome" to activate'],
|
||||
check=True, capture_output=True, timeout=5,
|
||||
)
|
||||
return FocusResult(True, "activated via osascript")
|
||||
except Exception as exc:
|
||||
return FocusResult(False, f"could not activate Chrome ({exc})")
|
||||
|
||||
running = {a.bundleIdentifier(): a for a in ws.runningApplications()}
|
||||
for bundle in MAC_BUNDLES:
|
||||
app = running.get(bundle)
|
||||
if app is None:
|
||||
continue
|
||||
# NSApplicationActivateIgnoringOtherApps — take focus even though the
|
||||
# terminal currently owns it.
|
||||
app.activateWithOptions_(1 << 1)
|
||||
return FocusResult(True, f"activated {bundle}")
|
||||
|
||||
return FocusResult(False, "no Chrome-family browser is running")
|
||||
|
||||
|
||||
def _other_activate() -> FocusResult:
|
||||
"""Windows/Linux: pyautogui already depends on pygetwindow, so use it."""
|
||||
try:
|
||||
import pygetwindow
|
||||
except ImportError:
|
||||
return FocusResult(False, "pygetwindow unavailable — cannot raise the browser")
|
||||
|
||||
wins = [w for w in pygetwindow.getAllWindows()
|
||||
if w.title and "Chrome" in w.title and w.visible]
|
||||
if not wins:
|
||||
return FocusResult(False, "no Chrome window found")
|
||||
try:
|
||||
win = wins[0]
|
||||
if getattr(win, "isMinimized", False):
|
||||
win.restore()
|
||||
win.activate()
|
||||
return FocusResult(True, f"activated window {win.title[:40]!r}")
|
||||
except Exception as exc:
|
||||
return FocusResult(False, f"could not activate window ({exc})")
|
||||
|
||||
|
||||
def activate_browser() -> FocusResult:
|
||||
"""Raise the browser application above everything else."""
|
||||
result = _mac_activate() if sys.platform == "darwin" else _other_activate()
|
||||
if result.ok:
|
||||
time.sleep(SETTLE)
|
||||
return result
|
||||
|
||||
|
||||
def ensure_frontmost(timeout: float = 1.5) -> FocusResult:
|
||||
"""Raise the browser and, where we can check, confirm it actually came forward.
|
||||
|
||||
Returning ok=False does not mean the click will fail — only that we could not
|
||||
verify. The caller decides whether to proceed.
|
||||
"""
|
||||
result = activate_browser()
|
||||
if not result.ok:
|
||||
return result
|
||||
|
||||
if sys.platform != "darwin":
|
||||
return result # no cheap way to verify; activation call succeeded
|
||||
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
front = frontmost()
|
||||
if front is None:
|
||||
return FocusResult(True, result.detail + " (unverified)")
|
||||
if front in MAC_BUNDLES:
|
||||
return FocusResult(True, f"{front} is frontmost")
|
||||
time.sleep(0.05)
|
||||
|
||||
return FocusResult(False, f"browser did not come to the front (frontmost is {frontmost()})")
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Human-like cursor motion for pyautogui.
|
||||
|
||||
A straight-line teleport followed by an instant click is not just conspicuous — it
|
||||
is unreliable. Plenty of web UIs only arm a control once it has actually been
|
||||
hovered (dropdowns, custom widgets, tooltip-gated buttons), and a cursor that
|
||||
arrives and presses in the same tick can beat the page's own mousemove handlers.
|
||||
|
||||
So the motion here does what a hand does: accelerates out, coasts, decelerates in
|
||||
along a slightly curved path, occasionally overshoots and corrects, pauses a beat
|
||||
before pressing, and holds the button down for a human interval.
|
||||
"""
|
||||
|
||||
import math
|
||||
import random
|
||||
import time
|
||||
|
||||
import pyautogui
|
||||
|
||||
# pyautogui sleeps PAUSE seconds after *every* call. With a stepped path that
|
||||
# would add tens of seconds, so we take over timing entirely.
|
||||
pyautogui.PAUSE = 0
|
||||
|
||||
# Motion feel. Durations in seconds, distances in pixels.
|
||||
MIN_DURATION = 0.15
|
||||
MAX_DURATION = 1.70
|
||||
CURVE_STRENGTH = 0.18 # lateral bow, as a fraction of travel distance
|
||||
TREMOR = 0.7 # sub-pixel hand tremor
|
||||
OVERSHOOT_ABOVE = 260.0 # only long throws overshoot
|
||||
OVERSHOOT_CHANCE = 0.55
|
||||
DWELL = (0.06, 0.17) # settle after arriving, before pressing
|
||||
HOLD = (0.055, 0.12) # how long the button stays down
|
||||
|
||||
|
||||
def _ease(t: float) -> float:
|
||||
"""Smootherstep: zero velocity at both ends, quick through the middle."""
|
||||
return t * t * t * (t * (t * 6 - 15) + 10)
|
||||
|
||||
|
||||
def _bezier(p0, p1, p2, p3, t):
|
||||
"""Cubic Bezier — the bow that keeps the path off a dead-straight line."""
|
||||
u = 1 - t
|
||||
return (
|
||||
u * u * u * p0[0] + 3 * u * u * t * p1[0] + 3 * u * t * t * p2[0] + t * t * t * p3[0],
|
||||
u * u * u * p0[1] + 3 * u * u * t * p1[1] + 3 * u * t * t * p2[1] + t * t * t * p3[1],
|
||||
)
|
||||
|
||||
|
||||
def _duration_for(distance: float, rng: random.Random) -> float:
|
||||
"""Farther costs more, but sub-linearly — pointing time grows roughly with the
|
||||
square root of distance over the range a screen covers. Calibrated so a nudge
|
||||
of ~40px takes ~0.2s and a throw across a large display takes ~0.8s; a log
|
||||
curve here would spend a full second creeping 40 pixels."""
|
||||
base = 0.10 + 0.018 * math.sqrt(distance)
|
||||
return max(MIN_DURATION, min(MAX_DURATION, base * rng.uniform(0.85, 1.2)))
|
||||
|
||||
|
||||
def _glide(start, end, duration: float, rng: random.Random) -> None:
|
||||
"""One curved, eased sweep from start to end."""
|
||||
dx, dy = end[0] - start[0], end[1] - start[1]
|
||||
distance = math.hypot(dx, dy)
|
||||
if distance < 1:
|
||||
return
|
||||
|
||||
# Control points pushed perpendicular to the direction of travel, so the path
|
||||
# bows to one side the way an arm swings rather than tracking a ruler.
|
||||
nx, ny = -dy / distance, dx / distance
|
||||
bow = distance * CURVE_STRENGTH * rng.uniform(-1, 1)
|
||||
c1 = (start[0] + dx * 0.3 + nx * bow, start[1] + dy * 0.3 + ny * bow)
|
||||
c2 = (start[0] + dx * 0.7 + nx * bow * rng.uniform(0.4, 1.0),
|
||||
start[1] + dy * 0.7 + ny * bow * rng.uniform(0.4, 1.0))
|
||||
|
||||
steps = max(14, min(95, int(distance / 5)))
|
||||
step_time = duration / steps
|
||||
next_at = time.perf_counter()
|
||||
|
||||
for i in range(1, steps + 1):
|
||||
t = _ease(i / steps)
|
||||
x, y = _bezier(start, c1, c2, end, t)
|
||||
|
||||
# Tremor fades out as we close in, so the landing stays accurate.
|
||||
if i < steps:
|
||||
decay = 1 - (i / steps)
|
||||
x += rng.gauss(0, TREMOR) * decay
|
||||
y += rng.gauss(0, TREMOR) * decay
|
||||
|
||||
pyautogui.moveTo(x, y, duration=0, _pause=False)
|
||||
|
||||
next_at += step_time
|
||||
slack = next_at - time.perf_counter()
|
||||
if slack > 0:
|
||||
time.sleep(slack)
|
||||
|
||||
|
||||
def move(x: float, y: float, rng: random.Random | None = None) -> None:
|
||||
"""Move the cursor to (x, y) the way a hand would."""
|
||||
rng = rng or random.Random()
|
||||
start = pyautogui.position()
|
||||
distance = math.hypot(x - start[0], y - start[1])
|
||||
if distance < 1:
|
||||
return
|
||||
|
||||
duration = _duration_for(distance, rng)
|
||||
|
||||
# A long throw usually lands slightly past the mark and gets pulled back.
|
||||
if distance > OVERSHOOT_ABOVE and rng.random() < OVERSHOOT_CHANCE:
|
||||
angle = math.atan2(y - start[1], x - start[0]) + rng.uniform(-0.35, 0.35)
|
||||
past = rng.uniform(6, 16)
|
||||
overshoot = (x + math.cos(angle) * past, y + math.sin(angle) * past)
|
||||
_glide(start, overshoot, duration * 0.82, rng)
|
||||
time.sleep(rng.uniform(0.02, 0.06))
|
||||
_glide(pyautogui.position(), (x, y), rng.uniform(0.10, 0.19), rng)
|
||||
else:
|
||||
_glide(start, (x, y), duration, rng)
|
||||
|
||||
# Land exactly on target — accumulated float error must not cost us the click.
|
||||
pyautogui.moveTo(x, y, duration=0, _pause=False)
|
||||
|
||||
|
||||
def click(x: float, y: float, rng: random.Random | None = None, press: bool = True) -> None:
|
||||
"""Move to (x, y), settle, then press and release.
|
||||
|
||||
With press=False the cursor travels and dwells but no button event is sent.
|
||||
"""
|
||||
rng = rng or random.Random()
|
||||
move(x, y, rng)
|
||||
|
||||
# A beat between arriving and pressing: this is what lets hover handlers,
|
||||
# CSS transitions and lazily-armed controls catch up before the press.
|
||||
time.sleep(rng.uniform(*DWELL))
|
||||
if not press:
|
||||
return
|
||||
|
||||
pyautogui.mouseDown(_pause=False)
|
||||
time.sleep(rng.uniform(*HOLD))
|
||||
pyautogui.mouseUp(_pause=False)
|
||||
@@ -0,0 +1,6 @@
|
||||
pyautogui>=0.9.54
|
||||
# macOS backends: Quartz drives the mouse, Cocoa (AppKit) raises the browser
|
||||
# window without needing AppleScript automation permission.
|
||||
pyobjc-core>=10.0; sys_platform == "darwin"
|
||||
pyobjc-framework-Quartz>=10.0; sys_platform == "darwin"
|
||||
pyobjc-framework-Cocoa>=10.0; sys_platform == "darwin"
|
||||
Reference in New Issue
Block a user