Playwright vs Puppeteer for Web Scraping

The short answer

Use Playwright for new scraping projects. One API across Chromium, Firefox and WebKit; auto-waiting that removes most of the flakiness; first-class network interception; and official bindings for Python, .NET and Java, not just Node.

Keep Puppeteer if you already have a working Chromium-only codebase and a small dependency budget. It is not worse software — it is narrower software, and migrating a functioning crawler for the sake of it is a poor use of a sprint.

The rest of this article is the detail behind that, plus the part that matters more than the choice itself.

Auto-waiting is the difference you feel daily

Puppeteer's model is "do the thing now". Playwright's is "wait until doing the thing makes sense, then do it" — attached to the DOM, visible, stable (not animating), enabled, and not obscured by another element. In scraping, where you're driving pages you didn't build and cannot instrument, that difference removes an entire category of bug.

The practical symptom is `sleep`. Every mature Puppeteer scraper accumulates arbitrary waits that someone added at 2am to fix a flaky selector, and they are simultaneously too short on a slow proxy and pure wasted latency on a fast one. At 100k pages a day, two seconds of defensive sleep per page is 55 hours of paid browser time.

await page.goto(url, { waitUntil: "networkidle2" });
await page.waitForSelector(".price", { visible: true });
await new Promise((r) => setTimeout(r, 1500));  // "it's flaky otherwise"
const price = await page.$eval(".price", (el) => el.textContent.trim());

await page.goto(url, { waitUntil: "domcontentloaded" });
// Waits for attached + visible + stable, retries until timeout, then asserts.
const price = (await page.locator(".price").first().innerText()).trim();

Intercept the JSON instead of parsing the DOM

This is the technique that changes scraper economics, and Playwright makes it pleasant. Most modern pages fetch their data as JSON and then render it. If you capture the JSON, you get typed, complete, unpaginated data — including fields the UI never displays — and you become immune to every CSS change the frontend team ships.

Both libraries can do it. Playwright's `route`/`response` API is cleaner, and its `waitForResponse` predicate makes the "click, then grab the resulting call" pattern a two-liner.

const products = [];

// Passive capture: watch every response the page makes.
page.on("response", async (res) => {
  if (!res.url().includes("/api/products")) return;
  if (!res.headers()["content-type"]?.includes("json")) return;
  const json = await res.json().catch(() => null);
  if (json?.items) products.push(...json.items);
});

// Active: trigger a fetch and await exactly that response.
const [listing] = await Promise.all([
  page.waitForResponse((r) => r.url().includes("/api/search") && r.ok()),
  page.getByRole("button", { name: "Next page" }).click(),
]);
const page2 = await listing.json();

Once you've found the JSON endpoint, ask whether you still need the browser at all. Very often you can call it directly with the right headers and drop your cost per page by an order of magnitude.

Block what you don't need

A product page pulls images, fonts, analytics beacons, chat widgets and ad tags. You want none of them. Blocking them typically cuts page load time by half or better and cuts bandwidth by far more — and bandwidth is what proxy bills are made of.

Be careful with one thing: blocking too aggressively is itself a detection signal, and some sites genuinely need their own scripts to render content. Block third-party media and trackers, not the site's own JS.

const BLOCKED_TYPES = new Set(["image", "media", "font"]);
const BLOCKED_HOSTS = [/doubleclick/, /google-analytics/, /hotjar/, /intercom/];

await context.route("**/*", (route) => {
  const req = route.request();
  const url = req.url();
  if (BLOCKED_TYPES.has(req.resourceType())) return route.abort();
  if (BLOCKED_HOSTS.some((re) => re.test(url))) return route.abort();
  return route.continue();
});

Contexts, not browsers, are your unit of isolation

The most common performance mistake in both libraries is launching a browser per job. A Chromium process costs hundreds of milliseconds and a few hundred megabytes to start. A browser context costs almost nothing and gives you a clean cookie jar, storage and cache.

One browser, N contexts, one page per context, contexts recycled between logical sessions. That maps neatly onto proxy session strategy too: one context per identity, so cookies and IP rotate together instead of drifting apart — which is a detection signal in itself.

const browser = await chromium.launch();

async function withSession(sessionId, fn) {
  const context = await browser.newContext({
    userAgent: UA,
    locale: "de-DE",
    timezoneId: "Europe/Berlin",
    viewport: { width: 1440, height: 900 },
    proxy: {
      server: "http://proxy.your-provider.com:8080",
      username: `customer-acme-country-de-session-${sessionId}`,
      password: process.env.PROXY_KEY,
    },
  });
  try {
    return await fn(await context.newPage());
  } finally {
    await context.close();   // identity retired with the session
  }
}

Python, and why it decides a lot of teams

Puppeteer is Node. The Python port (`pyppeteer`) has been effectively unmaintained for years and you should not start anything on it. Playwright ships official, supported Python bindings with both sync and async APIs.

If your data stack is pandas, Airflow and dbt — which for most scraping teams it is — that alone settles the argument. Keeping the crawler in the same language as the pipeline that consumes it is worth more than any API nicety.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    ctx = browser.new_context(locale="en-GB", timezone_id="Europe/London")
    page = ctx.new_page()
    page.goto("https://example.com/catalog", wait_until="domcontentloaded")
    rows = page.locator("[data-product-id]")
    for i in range(rows.count()):
        print(rows.nth(i).get_attribute("data-product-id"))
    browser.close()

A fair word for Puppeteer

It's smaller, it starts faster, its dependency tree is shorter, and if you only ever target Chromium you are not missing much. The Chrome DevTools team's proximity to it means new CDP surfaces sometimes land there first. For a focused, single-site, Node-only crawler that already works, Puppeteer is a perfectly respectable answer and "we should migrate to Playwright" is not a business case on its own.

The part neither one solves

Both libraries drive a browser. Neither gives you clean IPs, TLS and HTTP/2 fingerprint coherence, challenge handling, autoscaling, per-domain concurrency budgets, session persistence across restarts, or the on-call rota when Chromium's memory grows overnight and the pod OOMs at 4am. That list is where essentially all of the real cost of scraping lives, and choosing between two excellent libraries does nothing about any of it.

Running your own farm is genuinely fine at small scale. Past a few hundred thousand sessions a month, handing the fetch-and-render step to an API usually costs less than the engineering time it replaces, and you keep your parsing code exactly as it is.

// Or skip the browser entirely: Irmu renders the page for you.
const params = new URLSearchParams({
  url: "https://example.de/catalog",
  js: "true",
  country: "de",
});

const res = await fetch(`https://app.irmu.com/api/crawl?${params}`, {
  headers: { Authorization: `Bearer ${process.env.IRMU_API_KEY}` },
});

const html = await res.text();

Decide it in thirty seconds

New project, or any Python involved, or you need Firefox/WebKit: Playwright. Existing Node-only Chromium crawler that works: leave it. Either way, spend the time you saved on interception, resource blocking and session strategy — those move your cost and reliability far more than the library name does.