Zillow API Alternative: Build Your Own Real Estate Database

There is no Zillow API you can sign up for

This is the part nobody says plainly. Zillow's public developer program has been effectively closed for years: the old GetSearchResults and GetZestimate endpoints were retired, and what remains is a partner program aimed at brokerages, MLS members and syndication partners. If you are a two-person team building a rental analytics tool, you will not be issued a key.

So when people search for a Zillow API what they usually need is a property data pipeline: a repeatable way to turn public listing pages into rows in a database, with fields that stay consistent as the site changes. That is a solvable problem, and the rest of this post is the shape of the solution.

Before the code, the honest caveats. Zillow's terms of use prohibit automated collection, and the site is behind PerimeterX. Listing content and photos are copyrighted, and MLS data carries its own redistribution rules. Nothing here is legal advice: many teams crawl public pages for internal research and comparables, far fewer can lawfully republish them. Decide which one you are before you build.

If your use case is a consumer-facing listings site, license the data. Crawling public pages is a reasonable path for internal analysis, market research and comparables — it is a bad path for redistribution.

What a listing page actually gives you

Modern real-estate portals render their detail pages client-side and hydrate from a JSON blob embedded in the document. That blob is where the good fields live — far more than what the visible page shows. A single detail page typically carries price, beds, baths, interior square footage, lot size, year built, property type, tax history, price history, HOA fees, days on market, agent and brokerage, latitude and longitude, and a photo manifest.

The practical consequence: you want the fully rendered document, not the initial HTML shell, and you want to pull fields by meaning rather than by CSS selector. Selectors on these sites are generated class names that change between deploys and differ across A/B buckets. A field description survives a redesign; `.sc-1x2y3z .price` does not.

• Identity: address, unit, ZIP, latitude, longitude, parcel or listing ID.
• Money: list price, price per square foot, price history with dates, taxes, HOA.
• Physical: beds, baths, interior area, lot area, year built, property type, parking.
• Market: status, days on market, listing date, agent, brokerage.

One page, one call

Irmu exposes a single endpoint, `GET /crawl`. You give it a URL and it returns the page after JavaScript has run. Two switches matter for real estate: `premium` routes the request through residential IPs instead of datacenter ones, which is what gets you past PerimeterX on the large portals, and `ai_query` answers a question about the page so you get fields back instead of a wall of markup.

Start by proving a single page works before you build anything around it:

curl -G https://app.irmu.com/api/crawl \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  --data-urlencode "url=https://www.zillow.com/homedetails/example_zpid/" \
  --data-urlencode "premium=true" \
  --data-urlencode "js=true" \
  --data-urlencode "ai_query=Return JSON with keys address, city, state, zip, price_usd, beds, baths, sqft, lot_sqft, year_built, property_type, status, days_on_market, hoa_monthly, latitude, longitude. Use null for anything not stated on the page."

`js` defaults to on. Turning it off costs less but returns the pre-hydration shell, which on a listing portal is an empty skeleton. Keep it on for detail pages; turn it off only for sitemaps and other static documents.

Discovering listing URLs without a search API

The awkward part of any listings crawl is not the detail page, it is finding the detail pages. Search result pages are paginated, personalized and often capped at a few hundred results per query — which means a naive "all homes in Texas" crawl silently truncates.

Two techniques get you full coverage. First, partition your search by geography until every partition fits under the result cap: state, then metro, then ZIP, then price band if a ZIP is still too dense. Second, check the portal's XML sitemaps, which are public, static, and enumerate detail URLs without rendering anything — a sitemap fetch with `js=false` costs a single credit.

Whatever the source, dedupe on the portal's own listing ID rather than the URL. The same property is reachable through several URL shapes, and you do not want to pay to crawl it four times.

import re
import requests

API = "https://app.irmu.com/api/crawl"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

def fetch(url, *, js=True, premium=False, ai_query=None):
    params = {"url": url, "js": "true" if js else "false"}
    if premium:
        params["premium"] = "true"
    if ai_query:
        params["ai_query"] = ai_query
    r = requests.get(API, headers=HEADERS, params=params, timeout=180)
    r.raise_for_status()
    return r.text

def listing_urls_from_sitemap(sitemap_url):
    # Sitemaps are static XML: no rendering needed, so this costs 1 credit.
    xml = fetch(sitemap_url, js=False)
    return re.findall(r"<loc>([^<]+)</loc>", xml)

def zpid(url):
    m = re.search(r"/(\d+)_zpid", url)
    return m.group(1) if m else url

urls = listing_urls_from_sitemap("https://www.zillow.com/xml/indexes/us/hdp/for-sale.xml.gz")
unique = {zpid(u): u for u in urls}
print(len(unique), "unique listings discovered")

From pages to rows

Now the loop. Keep it boring: bounded concurrency, retries with backoff, and every raw response written to disk before you parse it. That last habit pays for itself the first time you realize your prompt was missing a field — you re-parse from local files instead of paying to crawl 40,000 pages again.

Ask for JSON explicitly and validate what comes back. A language model reading a page is dramatically more resilient than selectors, but it is not a parser: pin the key names, demand nulls for missing values, and reject rows that fail a sanity check rather than writing `beds: "three"` into an integer column.

import json, pathlib, time
from concurrent.futures import ThreadPoolExecutor

SCHEMA_PROMPT = (
    "Extract this property listing as JSON with exactly these keys: "
    "listing_id, address, city, state, zip, price_usd, beds, baths, sqft, "
    "lot_sqft, year_built, property_type, status, days_on_market, "
    "hoa_monthly, latitude, longitude, agent_name, brokerage. "
    "Numbers must be numbers, not strings. Use null when the page does not state a value. "
    "Return only the JSON object."
)

RAW = pathlib.Path("raw"); RAW.mkdir(exist_ok=True)

def scrape_listing(listing_id, url, attempt=1):
    cache = RAW / f"{listing_id}.json"
    if cache.exists():
        return json.loads(cache.read_text())
    try:
        body = fetch(url, js=True, premium=True, ai_query=SCHEMA_PROMPT)
        row = json.loads(body[body.index("{"): body.rindex("}") + 1])
    except Exception as exc:
        if attempt >= 3:
            print("giving up on", url, exc)
            return None
        time.sleep(2 ** attempt)
        return scrape_listing(listing_id, url, attempt + 1)
    row["listing_id"] = row.get("listing_id") or listing_id
    row["source_url"] = url
    cache.write_text(json.dumps(row))
    return row

def valid(row):
    if not row or not row.get("address"):
        return False
    price = row.get("price_usd")
    return isinstance(price, (int, float)) and 5_000 < price < 100_000_000

with ThreadPoolExecutor(max_workers=8) as pool:
    rows = list(pool.map(lambda kv: scrape_listing(*kv), unique.items()))

clean = [r for r in rows if valid(r)]
print(f"{len(clean)} of {len(rows)} rows passed validation")

const API = "https://app.irmu.com/api/crawl";

async function crawl(url, { js = true, premium = false, aiQuery } = {}) {
  const params = new URLSearchParams({ url, js: String(js) });
  if (premium) params.set("premium", "true");
  if (aiQuery) params.set("ai_query", aiQuery);

  const res = await fetch(`${API}?${params}`, {
    headers: { Authorization: `Bearer ${process.env.IRMU_API_KEY}` },
  });
  if (!res.ok) throw new Error(`crawl failed: ${res.status}`);
  return res.text();
}

const body = await crawl("https://www.zillow.com/homedetails/example_zpid/", {
  premium: true,
  aiQuery: "Return JSON: address, price_usd, beds, baths, sqft, year_built. Nulls where unknown.",
});
console.log(JSON.parse(body.slice(body.indexOf("{"), body.lastIndexOf("}") + 1)));

The database, and keeping it current

A property database is only interesting over time. The single most valuable thing you will own is not the current snapshot — anyone can see today's price on the site — it is the history: when a listing appeared, every price change, how long it sat, whether it went pending and came back.

Model that with two tables. One row per property holding the slowly-changing physical facts, and an append-only observations table holding one row per crawl. Never update a price in place; insert an observation and let the history accumulate. Re-crawl active listings on a cadence — daily for a watchlist, weekly for a metro — and diff each observation against the previous one to emit events.

create table properties (
  listing_id    text primary key,
  address       text not null,
  city          text,
  state         text,
  zip           text,
  latitude      double precision,
  longitude     double precision,
  beds          numeric,
  baths         numeric,
  sqft          integer,
  lot_sqft      integer,
  year_built    integer,
  property_type text,
  source_url    text,
  first_seen    timestamptz not null default now(),
  last_seen     timestamptz not null default now()
);

create table observations (
  id             bigserial primary key,
  listing_id     text references properties(listing_id),
  observed_at    timestamptz not null default now(),
  price_usd      numeric,
  status         text,
  days_on_market integer,
  hoa_monthly    numeric
);

create index on observations (listing_id, observed_at desc);

-- Every price change, straight out of the history.
select listing_id, observed_at, price_usd,
       price_usd - lag(price_usd) over w as delta
from observations
window w as (partition by listing_id order by observed_at)
order by observed_at desc;

What it costs

Irmu bills in credits, and the multipliers are published: a plain fetch is 1 credit, a rendered fetch is 5, a premium fetch is 10, a rendered premium fetch is 25, and an `ai_query` adds 5. So a full-fat listing crawl — residential IP, JavaScript rendered, fields extracted — is 30 credits per page. Geotargeting is free.

Run the arithmetic before you run the crawl. The $40 Lite plan carries 200,000 credits, which is about 6,600 fully-extracted listings a month, or roughly 40,000 if you skip the AI extraction and parse the embedded JSON yourself. The 200 free credits on signup are enough to validate the whole pipeline end to end on a handful of pages.

One billing detail worth internalizing: a 404, a 500 or an anti-bot block is still a served response, so it counts. What does not count is a failure on Irmu's side, or a domain we do not cover, such as `.gov`. That makes discovery cost real — filtering dead URLs out of your sitemap set before the expensive rendered pass is a genuine saving, not a micro-optimization.

• Discovery via sitemaps, `js=false`: 1 credit per document.
• Detail page, rendered, premium, no AI: 25 credits — cheapest if you parse the embedded JSON yourself.
• Detail page, rendered, premium, with `ai_query`: 30 credits — the resilient option.
• Daily refresh of a 2,000-property watchlist at 30 credits: 60,000 credits per day.

Where teams get this wrong

Three failure modes account for almost every abandoned real-estate crawler. Crawling search pages instead of partitioning them, and never noticing the result cap silently truncated the dataset. Storing only the current price, which throws away the only data that was actually scarce. And writing selectors, which means the pipeline breaks on a Tuesday when someone ships a redesign.

The fourth is scale for its own sake. A tight, current database of 5,000 properties in two metros beats a stale scrape of two million rows for nearly every real use case — and it costs about fifteen dollars a month to keep fresh.