How to Scrape Idealista and Other European Property Portals

European portals are a different problem

American real-estate crawling is largely a scale problem: a handful of huge portals, one language, one currency, one address format. Europe is a fragmentation problem. Every country has its own dominant portal, its own field conventions, its own consent regime, and often its own idea of what a bedroom is.

Idealista is the leader across Spain, Portugal and Italy. Fotocasa runs second in Spain, ImmobilienScout24 dominates Germany, SeLoger France, Rightmove and Zoopla the UK, Funda the Netherlands. They share almost no structure. What they do share is a common set of obstacles, and once you have solved those once, adding the next country is mostly a matter of a new field mapping.

• Access is geography-bound: several portals degrade or block requests from foreign IPs outright.
• GDPR consent walls sit in front of the content and must be dealt with before the DOM is useful.
• Areas are in square metres, prices in euros or pounds, and dates are day-first.
• Idealista in particular is aggressive with automated traffic and rate-limits hard.

Legality first, because it is different here

European law gives you a firmer footing in some respects and a much narrower one in others. Public listing facts — price, size, location — are generally not protected by copyright, and the EU database directive protects the collection rather than individual facts. Descriptions and photographs are protected, and republishing them is a straightforward infringement.

GDPR is the real constraint. Agent names, phone numbers and email addresses on a listing page are personal data, and collecting them puts you squarely under the regulation: you need a lawful basis, and the data subject retains rights over it. The simplest safe posture is to not collect contact fields at all unless your use case genuinely requires them, in which case get advice first.

Property facts: usually fine to collect for analysis. Descriptions and photos: copyrighted, do not republish. Agent contact details: personal data under GDPR, treat with care or skip entirely.

Getting a usable page back

Two switches carry most of the weight. `premium` puts the request on residential IPs, which is what these portals expect from a real visitor and what gets you past the block pages. `js` is on by default and you want it on for listing pages, since the content hydrates client-side.

The consent wall is worth understanding rather than fighting. On most of these sites the content is present in the DOM behind the overlay, so a rendered fetch already contains what you need and the banner is just noise an extraction step ignores. Where content is genuinely gated, the interstitial itself is what you will get back — which is why you always assert on a known field before writing a row, instead of trusting a 200.

curl -G https://app.irmu.com/api/crawl \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  --data-urlencode "url=https://www.idealista.com/inmueble/00000000/" \
  --data-urlencode "premium=true" \
  --data-urlencode "js=true" \
  --data-urlencode "ai_query=Devuelve JSON con: precio_eur, metros_construidos, metros_utiles, habitaciones, banos, planta, ascensor, tipo_operacion, tipo_inmueble, barrio, municipio, provincia, referencia. Usa null si el dato no aparece. No incluyas datos de contacto del anunciante."

Ask in the page's own language

A detail that makes a real difference: phrase your extraction prompt in the language of the page. Spanish listings say `metros construidos` and `metros útiles` — built area and usable area — and those are genuinely different numbers that a prompt written in English tends to collapse into one `sqm` field. German listings distinguish `Wohnfläche` from `Nutzfläche` and carry `Kaltmiete` versus `Warmmiete`, which is the difference between rent and rent-plus-utilities. Flattening that distinction quietly corrupts every rent comparison you build on top.

Extract in the local vocabulary, then normalize in your own code where the mapping is explicit and reviewable.

import json, requests

CRAWL = "https://app.irmu.com/api/crawl"

PORTALS = {
    "idealista.com": {
        "prompt": (
            "Devuelve solo JSON: precio_eur, metros_construidos, metros_utiles, "
            "habitaciones, banos, planta, ascensor, municipio, provincia, referencia. "
            "null si no aparece. Sin datos de contacto."
        ),
        "map": {"metros_construidos": "area_built_sqm", "metros_utiles": "area_usable_sqm",
                "habitaciones": "beds", "banos": "baths", "precio_eur": "price"},
        "currency": "EUR",
    },
    "immobilienscout24.de": {
        "prompt": (
            "Gib nur JSON zurück: kaltmiete_eur, warmmiete_eur, kaufpreis_eur, "
            "wohnflaeche_qm, zimmer, baujahr, etage, stadt, bundesland. "
            "null wenn nicht angegeben. Keine Kontaktdaten."
        ),
        "map": {"wohnflaeche_qm": "area_built_sqm", "zimmer": "beds",
                "kaltmiete_eur": "rent_base", "warmmiete_eur": "rent_total"},
        "currency": "EUR",
    },
    "rightmove.co.uk": {
        "prompt": (
            "Return only JSON: price_gbp, bedrooms, bathrooms, floor_area_sqft, "
            "property_type, tenure, council_tax_band, town, postcode_outcode. "
            "null when absent. No agent contact details."
        ),
        "map": {"floor_area_sqft": "area_built_sqft", "bedrooms": "beds",
                "bathrooms": "baths", "price_gbp": "price"},
        "currency": "GBP",
    },
}

def portal_for(url):
    for host, cfg in PORTALS.items():
        if host in url:
            return cfg
    raise ValueError(f"no mapping for {url}")

def scrape(url, api_key):
    cfg = portal_for(url)
    res = requests.get(
        CRAWL,
        headers={"Authorization": f"Bearer {api_key}"},
        params={"url": url, "js": "true", "premium": "true", "ai_query": cfg["prompt"]},
        timeout=180,
    )
    res.raise_for_status()
    body = res.text
    raw = json.loads(body[body.index("{"): body.rindex("}") + 1])

    row = {cfg["map"][k]: v for k, v in raw.items() if k in cfg["map"]}
    if "area_built_sqft" in row and row["area_built_sqft"]:
        row["area_built_sqm"] = round(row.pop("area_built_sqft") * 0.092903, 1)
    row["currency"] = cfg["currency"]
    row["source_url"] = url
    return row

Cadence, coverage and rate limits

Idealista is the strictest of the group and rewards patience: modest concurrency, residential IPs, and a crawl spread across the day rather than fired in a burst. Treat a run of blocks as a signal to slow down, not as something to retry harder through — retries against a portal that has decided it does not like you are just credits converted into 403s.

For coverage, partition by administrative geography, which maps neatly onto how these portals structure their URLs. Province then municipality then district in Spain, Bundesland then Kreis in Germany, outcode in the UK. Search pages cap out, so keep subdividing until each partition fits under the cap, and dedupe on the portal's own reference number.

On refresh rate: the European market moves more slowly than the American one. Daily re-crawls of an entire city are usually waste. A weekly full pass plus a daily pass over the properties you are actively tracking captures nearly everything that matters at a fraction of the spend.

Cost, in euros

A rendered premium fetch is 25 credits and an `ai_query` adds 5, so a fully-extracted European listing costs 30 credits, exactly as it does anywhere else. There is no surcharge for geography — routing a request through Spain rather than Ohio costs nothing extra.

That puts a weekly pass over 3,000 Madrid listings at 90,000 credits a month, comfortably inside the $40 plan. A daily pass over the same set would be roughly 630,000, which is Standard territory at $90.

As always, a served block page or a 404 is a billed request; only Irmu-side failures and uncovered domains such as `.gov` are not. That is another argument for a slow, well-behaved crawl: on these portals, politeness is directly measurable on the invoice.

A short checklist before you scale up

Prove one listing per portal end to end, with a field assertion rather than a status check. Write the raw response to disk before parsing. Extract in the page's language, normalize in code. Skip agent contact data unless you have a reason and a basis for it. Partition geographically, dedupe on the portal reference, and keep an append-only history so you can answer the only question anyone will actually ask — what changed.