SmartScrape

Problem

Most "AI scraping" tools either ship as opaque SaaS with your data on someone else's servers, or require you to hand-author selectors for every site you want to track. Neither fits a freelance workflow where the same client might ask to monitor a competitor's price list one week and an in-stock RSS-less inventory page the next.

The brief was a tool that takes a URL and a plain-English description, produces a working scrape configuration, runs it on a schedule, detects what actually changed between runs, and notifies — all self-hosted, with the operator's own API keys.

SmartScrape screenshots

01 / 09 · Home — recent runs and at-a-glance status

Approach

A single Node/TypeScript service with a React SPA, deployed via Docker Compose alongside Postgres. The core loop is deliberately small:

• Configure. A wizard sends the target page and the user's brief to an LLM (OpenRouter / OpenAI / Anthropic — the user chooses) and proposes a JSON extraction schema, a comparison key, and notification rules. The user accepts, edits, or starts over manually.
• Fetch. Cheerio for static HTML where it works; Playwright when it doesn't. The choice is auto-detected per job.
• Diff. SHA-256 per extracted row, matched across runs by the user-chosen comparison key. The output is an added / removed / changed delta, not a full document diff.
• Notify. Rule types include any-change, new-items, removed-items, field threshold (price < 500), and field-value transitions. Delivery to email or Telegram.

Security work that mattered more than the AI surface: API keys and OAuth tokens encrypted with AES-256-GCM at rest, bcrypt-hashed passwords, JWT with rotating refresh tokens, an SSRF guard on every user-supplied URL, prompt-injection hardening on every extraction call, CORS locked to the configured frontend, Helmet headers, and zero outbound analytics.

Result

A working, deployable tool with mobile-responsive UI, OAuth-backed Google Sheets export, and a CI pipeline that gates merges. Built for the kind of freelance brief that comes in as a sentence — "tell me when this competitor drops a SKU below £X", or "flag new listings on this page overnight" — and turns into a working scrape config without hand-authoring selectors.

Honest status: engineering-complete at v0.9.0, not yet operated in production. The next step is a 60-day self-hosted pilot on my own infrastructure, running monitors I actually want — this page gets rewritten with real run counts, change deltas, and uptime once those numbers exist, not before.