Guides2026-08-0515 minIrmu Engineering

Scrape Amazon Product Data with Python

A practical walkthrough for pulling prices, buy box, stock and reviews from Amazon at scale.

The price you scrape is not 'the price'

This is the single most expensive misunderstanding in Amazon scraping, and it is worth dwelling on before any code. There is no canonical price for an ASIN. What the page shows depends on which seller currently holds the buy box, which delivery postcode the session resolves to, whether the session looks like a Prime member, and which of several concurrent layout experiments the visitor was bucketed into.

Concretely: scrape a US ASIN through an unpinned proxy pool and you will get a price from whichever country the exit node happened to be in, converted or not, with a buy box that may belong to a third-party seller who does not even ship to your customers. The data looks plausible. It is wrong. Repricing decisions get made on it.

So pin everything you can, and record what you pinned alongside the value. A price row without marketplace, postcode and timestamp is not a measurement, it's a rumour.

Store marketplace, delivery postcode, currency and capture timestamp on every price row. Future you will need them to explain a chart.

What Amazon serves, and how

The product page is a hybrid. Title, bullets, images and the main price block are largely server-rendered, which means a well-formed HTTP request can get them without a browser. Buy box details, delivery promise, coupon badges, variant matrices and the review carousel arrive later over XHR.

That split is your cost lever. If you only need title, price, rating and review count, you do not need to render — and a non-rendered fetch is a fraction of the cost and latency of a browser session. Render only for the fields that genuinely require it.

No render needed: title, brand, bullets, main image, list price, rating, review count.
Render usually needed: buy box seller, delivery date, coupons, variant availability matrix.
Render plus interaction: full review pages, Q&A, 'see all buying options'.

A single product, done properly

Start with one ASIN and get the boring parts right: a session with retries, an explicit timeout, a residential IP in the marketplace's country, and a schema so the response is typed rather than a bag of strings you regex later.

amazon.py

import os
from dataclasses import dataclass
from datetime import datetime, timezone

import requests

API = "https://app.irmu.com/api"
HEADERS = {"Authorization": f"Bearer {os.environ['IRMU_API_KEY']}"}

MARKETPLACES = {
    "us": ("amazon.com", "78701"),
    "uk": ("amazon.co.uk", "SW1A 1AA"),
    "de": ("amazon.de", "10115"),
}

SCHEMA = {
    "title": "string",
    "brand": "string",
    "price": "number",
    "list_price": "number|null",
    "currency": "string",
    "rating": "number",
    "review_count": "number",
    "buybox_seller": "string|null",
    "in_stock": "boolean",
    "delivery_text": "string|null",
}

@dataclass
class Snapshot:
    asin: str
    marketplace: str
    postcode: str
    captured_at: str
    data: dict

def product(asin: str, marketplace: str = "us", render: bool = True) -> Snapshot:
    domain, postcode = MARKETPLACES[marketplace]
    res = requests.post(
        f"{API}/extract",
        headers=HEADERS,
        json={
            "url": f"https://www.{domain}/dp/{asin}",
            "country": marketplace,
            "render": render,
            "premium": True,          # residential IP: buy box data needs one
            "locale": marketplace,
            "cookies": {"lc-main": marketplace, "zip": postcode},
            "schema": SCHEMA,
        },
        timeout=90,
    )
    res.raise_for_status()
    return Snapshot(
        asin=asin,
        marketplace=marketplace,
        postcode=postcode,
        captured_at=datetime.now(timezone.utc).isoformat(),
        data=res.json()["data"],
    )

snap = product("B0CHX1W1XY")
print(snap.data["price"], snap.data["currency"], "|", snap.data["buybox_seller"])

response

{
  "status": 200,
  "credits": 25,
  "data": {
    "title": "Anker 737 Power Bank (PowerCore 24K)",
    "brand": "Anker",
    "price": 109.99,
    "list_price": 149.99,
    "currency": "USD",
    "rating": 4.6,
    "review_count": 18422,
    "buybox_seller": "AnkerDirect",
    "in_stock": true,
    "delivery_text": "FREE delivery Tue, Sep 1"
  },
  "meta": { "rendered": true, "country": "us", "attempts": 1 }
}

Validate before you trust a 200

Amazon rarely blocks you with a clean error. It serves you a 200 containing a dog picture and "Sorry, we just need to make sure you're not a robot", or a stripped-down page with the price block missing. If your pipeline only checks the status code, those land in your warehouse as nulls and zeros.

Write the assertion you actually care about — required fields are present and sane — and treat a violation as a retry with a fresh session, not as data.

python

class SoftBlock(Exception):
    pass

def validate(d: dict) -> dict:
    if not d.get("title"):
        raise SoftBlock("no title — likely a captcha interstitial")
    if d.get("in_stock") and not d.get("price"):
        raise SoftBlock("in stock but no price — price block did not render")
    if d.get("price") and d["price"] <= 0:
        raise SoftBlock(f"implausible price {d['price']}")
    return d

def product_checked(asin: str, marketplace: str = "us", attempts: int = 3):
    last = None
    for i in range(attempts):
        try:
            return validate(product(asin, marketplace).data)
        except SoftBlock as e:
            last = e
            time.sleep(2 ** i)      # 1s, 2s, 4s — new session each time
    raise last

Every one of those retries is a delivered response, so every one is billed. Cap attempts at three and alert on the rate — a rising soft-block rate is a signal, not something to paper over with more retries.

Scaling to a catalog

Once one ASIN works, do not wrap it in a thread pool and point it at 50,000 more. You will hold 50,000 connections open, your process will own the retry state for all of them, and one deploy will lose the lot.

Submit batches instead and receive results by webhook as they complete. Your worker becomes stateless, the queue survives your restarts, and back-pressure is somebody else's problem.

batch.py

def submit(asins: list[str], marketplace: str = "us", chunk: int = 500):
    domain, _ = MARKETPLACES[marketplace]
    batches = []
    for i in range(0, len(asins), chunk):
        part = asins[i : i + chunk]
        res = requests.post(
            f"{API}/extract/batch",
            headers=HEADERS,
            json={
                "urls": [f"https://www.{domain}/dp/{a}" for a in part],
                "country": marketplace,
                "render": True,
                "premium": True,
                "schema": SCHEMA,
                "webhook": "https://hooks.example.com/irmu/amazon",
                "idempotency_key": f"catalog-{marketplace}-{i}-{date.today()}",
            },
            timeout=60,
        )
        res.raise_for_status()
        batches.append(res.json()["batch_id"])
    return batches

webhook_receiver.py (FastAPI)

@app.post("/irmu/amazon")
async def receive(payload: dict):
    for item in payload["results"]:
        if item["status"] != 200:
            await log_failure(item["url"], item["status"])
            continue
        try:
            row = validate(item["data"])
        except SoftBlock as e:
            await requeue(item["url"], reason=str(e))
            continue
        await upsert_price_history(
            asin=item["url"].rsplit("/", 1)[-1],
            price=row["price"],
            currency=row["currency"],
            buybox=row["buybox_seller"],
            captured_at=payload["completed_at"],
        )
    return {"ok": True}

Model prices as history, never as a column

The mistake that costs the most rework: storing `price` on the product row and overwriting it. You lose every trend, you cannot answer "when did they undercut us", and a single bad scrape silently corrupts the current value with no way to detect it.

Append-only price history with the capture context is boring and correct. Deduplicate on unchanged values if storage matters, but keep the first and last observation of each price level so you can reconstruct the change points.

sql

create table price_history (
  id            bigserial primary key,
  asin          text        not null,
  marketplace   text        not null,
  postcode      text        not null,
  price         numeric(12,2),
  currency      text        not null,
  buybox_seller text,
  in_stock      boolean     not null,
  captured_at   timestamptz not null default now()
);

create index on price_history (asin, marketplace, captured_at desc);

-- current price per ASIN, without destroying the trail
select distinct on (asin, marketplace) asin, marketplace, price, captured_at
from price_history
order by asin, marketplace, captured_at desc;

Reviews and content audits

Reviews are a separate crawl with a separate cadence. Ask for an array with text, rating, date, verified-purchase flag and variant, then do your own clustering to surface defect themes — "battery swells after 4 months" shows up in review text months before it shows up in returns data.

Content audits are the underrated one. Extract bullets, A+ modules and image URLs, diff them nightly against your PIM export, and you catch the cases where a marketplace edit or a rogue seller quietly rewrote your listing. That diff has paid for more scraping budgets than price monitoring has.

python

def reviews(asin: str, pages: int = 5, marketplace: str = "us"):
    domain, _ = MARKETPLACES[marketplace]
    out = []
    for page in range(1, pages + 1):
        res = requests.post(
            f"{API}/extract",
            headers=HEADERS,
            json={
                "url": f"https://www.{domain}/product-reviews/{asin}"
                       f"?sortBy=recent&pageNumber={page}",
                "country": marketplace,
                "render": True,
                "schema": {"reviews": [{
                    "rating": "number", "title": "string", "text": "string",
                    "date": "string", "verified": "boolean", "variant": "string|null",
                }]},
            },
            timeout=90,
        )
        batch = res.json()["data"]["reviews"]
        if not batch:
            break                      # ran past the last page
        out.extend(batch)
    return out

The short version

Pin the marketplace and postcode, and store them. Render only the fields that need it. Validate the payload rather than the status code. Batch and webhook instead of thread-pooling. Append price history, never overwrite it. And re-read Amazon's terms for your use case — public product data is one thing, and anything behind a login is another.

Keep reading

Guides

Start building with Irmu today

200 free credits every month, no card required. Every API, every integration, one key.

Start free Read the docs