# Replit web scraping API setup with Irmu

Scrape live web pages from a Replit app with Irmu: store your key in Replit Secrets, call the API from Python or Node, and handle slow jobs. Optional MCP for Replit Agent.

Store your Irmu key in Replit Secrets and call the API from your Repl's server code. If your Replit Agent supports custom MCP servers, connect Irmu so it can read pages while it builds.

## Option A: MCP server

> We have not confirmed the exact Replit menu for custom MCP servers. If you don't see the option, use the API steps below — they work in every Repl.

1. **Add a custom MCP server** — If Replit Agent offers custom MCP servers in your workspace, add one with the Irmu URL.

```text
https://app.irmu.com/mcp
```

2. **Authenticate** — Sign in with OAuth, or if the form asks for headers, add an Authorization header with your API key:

```json
{
  "mcpServers": {
    "irmu": {
      "url": "https://app.irmu.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_IRMU_API_KEY"
      }
    }
  }
}
```

## Option B: call the API from your app

1. **Add the secret** — Open Tools → Secrets in your Repl and add IRMU_API_KEY with a key from the Irmu dashboard.

2. **Python** — Install requests and call Irmu:

```python
import os, time, requests

API = "https://app.irmu.com/api"
HEADERS = {"Authorization": f"Bearer {os.environ['IRMU_API_KEY']}"}

def scrape(url: str) -> dict:
    res = requests.get(f"{API}/crawl", params={"url": url, "format": "markdown"}, headers=HEADERS)
    # Slow pages return 202 with a job id: poll until ready (polling is free).
    while res.status_code == 202:
        job = res.json()["job_id"]
        time.sleep(3)
        res = requests.get(f"{API}/crawl/{job}", headers=HEADERS)
    res.raise_for_status()
    return res.json()
```

3. **Node** — Or in Node 18+, with no extra packages:

```typescript
// Server-side only. Reads IRMU_API_KEY from the environment.
export async function scrape(url: string) {
  const key = process.env.IRMU_API_KEY;
  const params = new URLSearchParams({ url, format: "markdown" });
  let res = await fetch(`https://app.irmu.com/api/crawl?${params}`, {
    headers: { Authorization: `Bearer ${key}` },
  });
  // Slow pages return 202 with a job id: poll until ready (polling is free).
  while (res.status === 202) {
    const { job_id } = await res.json();
    await new Promise((r) => setTimeout(r, 3000));
    res = await fetch(`https://app.irmu.com/api/crawl/${job_id}`, {
      headers: { Authorization: `Bearer ${key}` },
    });
  }
  if (!res.ok) throw new Error(`Irmu error ${res.status}`);
  return res.json();
}
```

## Prompts to try

- "Write a Flask app that takes a URL, scrapes it with Irmu using IRMU_API_KEY, and returns the page as markdown." (tools: crawl)
- "Build a script that asks ChatGPT 'best CRM for startups' from the US through Irmu every week and logs which brands are named." (tools: chatgpt)
- "Create a bot that screenshots our homepage from Germany every morning with Irmu." (tools: screenshot)

## FAQ

**What does it cost?** The same credits as the REST API: 1 credit for a plain page, more with JavaScript rendering or premium IPs, 30 flat for Google AI Overview, ChatGPT and Booking.com. Using MCP adds nothing. See /pricing.

**I get a 401 error.** The API key is missing or wrong. Check the secret is named IRMU_API_KEY, has no extra spaces, and is read on the server, not in browser code.

**I get a 202 instead of data.** The page took longer than the request window. The response carries a job id; call GET /crawl/{job} until it stops returning 202. Collecting is free.

**Should I use MCP or the API?** MCP lets the AI agent fetch pages while it builds or answers you. The API is for the app you publish, so your users' actions trigger scrapes. Most projects use both.

Full MCP setup: https://irmu.com/docs/mcp · Recipe blocks: https://irmu.com/docs/recipes