# Browser Recipe blocks

Source: https://irmu.com/docs/recipes

Send a recipe as the JSON body of `POST https://app.irmu.com/api/crawl`. Fetch options (premium, country, city) go in the query string.

## The recipe

```json
{
  "name": "My recipe",
  "expectedOutput": "item",
  "blocks": [
    {
      "id": "b1",
      "type": "start",
      "details": {
        "type": "url",
        "source": "https://example.com/"
      }
    },
    {
      "id": "b2",
      "type": "extract",
      "details": {
        "name": "title",
        "selector": "h1",
        "property": "text"
      }
    }
  ]
}
```

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `name` | string | – | Optional label for your own logs. |
| `expectedOutput` | "item" \| "list" | "item" | How the result is returned. See How the result is built. |
| `blocks` (required) | array | – | Run in order, top to bottom. The first block must be a start block. |

Every block has the same shape:

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `id` (required) | string | – | Any string unique within the recipe. paginate refers to blocks by it. |
| `type` (required) | string | – | One of the block types below. |
| `details` (required) | object | – | The block's own fields. Use {} for blocks that take none. |

## How the result is built

The crawl fills a list of rows. `extract` writes match n into row n; every other data block writes to row 0.
With `expectedOutput: "list"` the rows are returned as an array. With `"item"`, row 0 is returned as one object; a field present in later rows becomes an array, and `url` is the start source.

## Pitfalls

- Don't send a top-level id on the recipe: the API rejects it with 422. Only name, expectedOutput and blocks are accepted.
- click and input fail the crawl when nothing matches; scroll-to-element, remove and extract skip instead.
- add-field and extract-variable need row 0 to exist; on an empty result the crawl fails.
- paginate only stops when a pass adds no rows, so include an extract inside the loop.

## `start`

Opens the page and gets past bot challenges. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `type` (required) | "url" | – | "url" is the only supported source. |
| `source` (required) | string | – | The URL to open. |

- Always the first block. Cloudflare, DDoS-Guard, EdgeOne, DataDome and PerimeterX challenges and captchas are handled automatically.
- A 404 ends the crawl with no result.
- A recipe with only a start block returns the raw HTML of the page instead of JSON.
- Only source is read from the start block. Page-load options (wait_until, js, adblock, premium, country, city) go in the POST /crawl query string.

```json
{"id":"b1","type":"start","details":{"type":"url","source":"https://example.com/"}}
```

## `wait`

Pauses for a number of seconds. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `seconds` (required) | number | – | How long to pause. Fractions are allowed (0.5). |


```json
{"id":"w1","type":"wait","details":{"seconds":2}}
```

## `click`

Clicks an element. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `selector` (required) | string | – | CSS selector. The first match is clicked. |

- If no element matches, the crawl fails. When the element might be missing, put a remove or wait before it.

```json
{"id":"c1","type":"click","details":{"selector":"button.load-more"}}
```

## `input`

Types text into an element. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `selector` (required) | string | – | CSS selector of the field to type into. |
| `text` (required) | string | – | Text to type. It is added to whatever the field already holds. |

- If no element matches, the crawl fails.

```json
{"id":"i1","type":"input","details":{"selector":"input[name=q]","text":"running shoes"}}
```

## `scroll-to-element`

Scrolls an element into view. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `selector` (required) | string | – | CSS selector. The first match is scrolled into view. |

- If nothing matches, the block is skipped. Useful for triggering lazy loading, for example by scrolling to the footer.

```json
{"id":"s1","type":"scroll-to-element","details":{"selector":"footer"}}
```

## `remove`

Removes elements from the page. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `selector` (required) | string | – | Every element it matches is removed. |
| `timeout` | number (ms) | 10000 | How long to wait for the selector. If it never appears, the block is skipped. |

- Use it to get rid of cookie banners and overlays before a click or screenshot.

```json
{"id":"r1","type":"remove","details":{"selector":"#cookie-banner"}}
```

## `paginate`

Goes to the next page and runs earlier blocks again. Writes: –.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startBlock` (required) | string | – | The id of the block to jump back to. It must exist and can't be the first block. |
| `selector` | string | – | The "next page" button. It is clicked, then the crawl waits 5 seconds. |

- Pagination stops when a full pass adds no new rows, so the loop must include an extract block.
- Use expectedOutput: "list".

```json
{"id":"next","type":"paginate","details":{"startBlock":"extract-title","selector":"a.next"}}
```

## `extract`

Reads a value from every element a selector matches. Writes: Row by row.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `name` (required) | string | – | The field to write. |
| `selector` (required) | string | – | CSS selector, looked up inside parentSelector if given, otherwise the whole page. |
| `property` (required) | string | – | text, html, an attribute (href, src, aria-label…) or dataset.<key>. Several can be comma-separated; the first with a value wins. |
| `parentSelector` | string | – | Defines the rows. A row that lacks the field stays empty and later rows keep their values. |
| `timeout` | number (ms) | 10000 | How long to wait for the selector (or parentSelector). If it never appears, the block is skipped. |

- A field named price longer than 20 characters is trimmed to the first currency amount found.

```json
{"id":"e1","type":"extract","details":{"name":"url","parentSelector":".card","selector":"a.title","property":"href"}}
```

## `jsonschema`

Reads a schema.org object from JSON-LD or microdata. Writes: Row 0 (replaces it).

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `type` (required) | string | – | The schema.org @type to find, such as Product, Hotel or Recipe. Not case-sensitive. |
| `timeout` | number (ms) | 30000 | How long to wait for JSON-LD before falling back to microdata. |

- Searches nested objects. If the JSON-LD object is sparse, missing fields are filled from microdata.

```json
{"id":"j1","type":"jsonschema","details":{"type":"Product"}}
```

## `datalayer`

Reads the product from the GTM / GA4 dataLayer. Writes: Row 0.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `type` (required) | "ecommerce" | – | ecommerce is the only supported value. |

- Takes the first product from ecommerce.detail.products (Universal Analytics) or ecommerce.items (GA4), plus currency.
- For GA4 only single-item events count and view_item_list is skipped, so listings aren't mistaken for a product.

```json
{"id":"d1","type":"datalayer","details":{"type":"ecommerce"}}
```

## `extract-variable`

Reads a JavaScript global. Writes: Row 0.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `variable` (required) | string | – | Dotted path from window, e.g. __NEXT_DATA__.props.pageProps. |
| `field` (required) | string | – | The field to write. |

- Needs row 0 to exist already: place it after extract or jsonschema.

```json
{"id":"v1","type":"extract-variable","details":{"variable":"__NEXT_DATA__.props.pageProps","field":"page"}}
```

## `add-field`

Adds a fixed value. Writes: Row 0.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `field` (required) | string | – | The field to write. |
| `value` (required) | any | – | The value, written as given. |

- Needs row 0 to exist already: place it after extract or jsonschema.

```json
{"id":"a1","type":"add-field","details":{"field":"source","value":"competitor-a"}}
```

## `screenshot`

Takes a full-page JPEG screenshot. Writes: data.screenshots.


- Takes no fields: "details": {}.
- Make it the last block of an "item" recipe: every screenshot the recipe took is returned as base64 in data.screenshots.

```json
{"id":"shot","type":"screenshot","details":{}}
```

## `ai`

Has your language model extract data or act on the page. Writes: Row 0.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `query` (required) | string | – | Plain-English instruction: data ("get the product details") or an action ("click on subscribe"). |

- Uses the language model and key saved in your organization settings (https://irmu.com/docs/dashboard). No key goes in the recipe.
- Expect POST /crawl to return a job id to poll with GET /crawl/{job} instead of the result.

```json
{"id":"ai1","type":"ai","details":{"query":"Get the product name, price and availability"}}
```

## `chatgpt`

Asks ChatGPT and captures the answer. Writes: Row 0 (conversation).

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `prompt` (required) | string | – | The question to ask. |

- Put it after a start block that opens https://chatgpt.com/ with JavaScript on. The answer is returned as structured nodes under conversation.

```json
{"id":"g1","type":"chatgpt","details":{"prompt":"What are the best web scraping APIs?"}}
```

## `booking`

Reads a Booking.com hotel page. Writes: Row 0.


- Takes no fields: "details": {}. Returns name, address, rating, price, rooms, facilities and images.
- Search result pages need extract blocks, or use GET /booking/search.

```json
{"id":"bk","type":"booking","details":{}}
```

## `google-ai-overview`

Reads Google's AI Overview. Writes: Row 0.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `field` | string | "ai_overview" | Field for the answer text. |
| `sourcesField` | string | "ai_overview_sources" | Field for the cited sources, as [{ title, url }]. |
| `timeout` | number (ms) | 15000 | How long to wait for the overview. |

- Accepts the consent dialog first if one appears. A page without an overview is left as it is.
- Or call GET /google-ai-overview directly.

```json
{"id":"aio","type":"google-ai-overview","details":{}}
```

## Full example

```json
{
  "name": "Product listing",
  "expectedOutput": "list",
  "blocks": [
    {
      "id": "open",
      "type": "start",
      "details": {
        "type": "url",
        "source": "https://example.com/shop"
      }
    },
    {
      "id": "cookies",
      "type": "remove",
      "details": {
        "selector": "#cookie-banner"
      }
    },
    {
      "id": "extract-title",
      "type": "extract",
      "details": {
        "name": "title",
        "parentSelector": ".card",
        "selector": "h3",
        "property": "text"
      }
    },
    {
      "id": "extract-price",
      "type": "extract",
      "details": {
        "name": "price",
        "parentSelector": ".card",
        "selector": ".price",
        "property": "text"
      }
    },
    {
      "id": "next",
      "type": "paginate",
      "details": {
        "startBlock": "extract-title",
        "selector": "a.next"
      }
    }
  ]
}
```