AI Web Scraping: Extract Data Without Writing Selectors

Why selectors keep breaking

Classic scrapers are a list of CSS selectors: `.price`, `h1.product-title`, `div[data-testid=rating]`. They're fast and cheap, and they break the moment the site ships a redesign, renames a class or hashes its CSS. On sites built with CSS-in-JS, the class names are random strings that change on every deploy.

AI web scraping flips the approach: instead of telling the scraper where the data is, you describe what you want and let a model find it in the rendered page. It costs more per page and it can be wrong, but it survives redesigns and works on sites you've never seen.

Option 1 — Ask a question with `ai_query`

The simplest form: add `ai_query` to a `GET /crawl`. Irmu fetches and renders the page as usual, then answers your question about it. The answer comes back as JSON in `data`. It costs 5 credits on top of the fetch, and an empty `ai_query` costs nothing extra.

Because the response switches from `text/html` to a JSON envelope when you ask a question, branch on the content type, not the status code.

curl -G https://app.irmu.com/api/crawl \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  --data-urlencode "url=https://shop.example/p/1" \
  --data-urlencode "ai_query=Return JSON with name, price, currency and in_stock (true/false)"

import os, requests

res = requests.get(
    "https://app.irmu.com/api/crawl",
    headers={"Authorization": f"Bearer {os.environ['IRMU_API_KEY']}"},
    params={
        "url": "https://shop.example/p/1",
        "ai_query": "Return JSON with name, price, currency and in_stock (true/false)",
    },
    timeout=200,
)
if res.headers.get("content-type", "").startswith("application/json"):
    print(res.json()["data"])
else:
    print("got HTML back:", res.text[:200])

Ask for an explicit shape. "Return JSON with name, price, currency" gets you parseable output; "what's on this page?" gets you prose.

Writing prompts that extract reliably

Treat `ai_query` like a function signature. The model has the whole page; your job is to remove ambiguity.

• Name every field and its type: `price (number, no currency symbol)`.
• Say what to do when something's missing: `use null if not shown`.
• Disambiguate: `the current price, not the crossed-out one`.
• For lists, ask for an array: `an array of {title, url} for every result`.
• Keep it under 1,000 characters — that's the limit.

Option 2 — Recipes: selectors first, AI as a fallback

For pages you scrape often, the cheapest reliable setup is a recipe on `POST /crawl`: structured data first, selectors for what's left, and AI only where markup is hostile. A `jsonschema` block reads JSON-LD and microdata, which most product, recipe and event pages publish and which survives redesigns. An `extract` block reads one selector. An `ai` block runs a free-form instruction.

Recipes cost a flat 30 credits. The body carries only `name`, `expectedOutput` and `blocks`; options like `premium` or `country` go in the query string.

{
  "name": "Product with fallback",
  "expectedOutput": "item",
  "blocks": [
    { "id": "b1", "type": "start", "details": { "type": "url", "source": "https://shop.example/p/1" } },
    { "id": "b2", "type": "jsonschema", "details": { "type": "Product" } },
    { "id": "b3", "type": "extract", "details": { "name": "stock", "selector": ".availability", "property": "text" } },
    { "id": "b4", "type": "ai", "details": { "query": "Return the shipping time as text, or null" } }
  ]
}

curl -X POST "https://app.irmu.com/api/crawl?country=us" \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  -H "Content-Type: application/json" \
  --data @recipe.json

Which one should you use?

A practical rule: `ai_query` for exploration, long-tail sites and one-off jobs; recipes for the handful of sites you hit every day.

• Unknown or many different sites → `ai_query`. No setup, +5 credits per page.
• A few sites, high volume, stable structured data → recipe with `jsonschema` and `extract`.
• Mostly stable, one messy field → recipe with an `ai` block just for that field.
• Lists across pages → recipe with `paginate` and `expectedOutput: "list"`.

Where AI extraction goes wrong

Models are confident even when they're wrong. The common failures are predictable: picking the crossed-out price, inventing a value that isn't on the page, mixing up two products on a comparison page, and returning the wrong number format. Validate every response against a schema, reject values that fail, and spot-check a sample against the page by hand.

And remember that AI only sees what the page served. If the target returned an anti-bot page, the model will happily answer questions about the challenge screen. That request is still a served, billed response. Check for empty or suspicious answers and retry with `premium=true` or a matching `country`.

from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    name: str
    price: float | None
    currency: str | None
    in_stock: bool | None

def parse(data):
    try:
        return Product.model_validate(data)
    except ValidationError as e:
        print("reject:", e.errors()[0]["msg"])
        return None

Getting started

Pick one page you scrape today, run it once with `ai_query`, and compare the output with your current scraper. If the AI version is right and the selectors keep breaking, that's your answer. If you need the same fields from the same site thousands of times a day, turn it into a recipe and keep AI for the parts that resist.