How to Rotate Residential Proxies Properly

Rotation is not the goal — coherence is

"Rotate every request" is the advice everyone starts with, and it is wrong more often than it is right. Think about what it looks like from the server: a user adds an item to a cart from São Paulo, views the cart from Warsaw, and checks delivery options from Detroit, all within eight seconds, carrying the same session cookie. No human does that. You have replaced one weak signal (repeat IP) with a very strong one (impossible travel).

The correct unit of rotation is the logical session — one IP for the duration of one coherent user journey, retired when that journey ends. Rotate between journeys, not within them.

If a request carries a cookie set by a previous request, it should go out from the same IP. That one rule prevents most session-level detection.

Sticky or per-request: pick by workload

Both modes are legitimate; they just fit different jobs. The question to ask is whether the target maintains state about you across requests.

• Per-request rotation: stateless, independent fetches. Product pages by ID, sitemap crawls, image downloads. No cookies, no order dependency.
• Sticky sessions: anything with a cart, a login, a multi-step search, a postcode selection, or pagination that uses a server-side cursor.
• Sticky with a short TTL: search-result crawling where you want continuity across pages but a fresh identity per query.

import uuid, requests

PROXY_HOST = "proxy.your-provider.com:8080"
KEY = os.environ["PROXY_KEY"]

def proxy(country: str, city: str | None = None, session: str | None = None):
    parts = ["customer-acme", f"country-{country}"]
    if city:
        parts.append(f"city-{city}")
    if session:
        parts.append(f"session-{session}")   # omit for per-request rotation
    user = "-".join(parts)
    url = f"http://{user}:{KEY}@{PROXY_HOST}"
    return {"http": url, "https": url}

# One identity for one journey
sid = uuid.uuid4().hex[:12]
s = requests.Session()
s.proxies = proxy("de", "berlin", session=sid)
s.headers["Accept-Language"] = "de-DE,de;q=0.9"

s.get("https://www.example.de/")                    # landing
s.get("https://www.example.de/search?q=kopfhoerer")  # search
s.get("https://www.example.de/product/42")           # detail — same IP throughout

Match the geography to the content

Prices, availability, language, tax display and sometimes the entire page layout vary by location. Collecting German prices through a US exit node doesn't just risk detection — it silently gives you the wrong numbers, which is worse, because a block is visible and a wrong price is not.

Coherence again: the IP's country, the `Accept-Language` header, the browser locale, the timezone and any currency or postcode cookie all have to tell the same story. A Berlin IP announcing `en-US` with an America/New_York timezone is a more interesting anomaly than a plain datacenter IP.

curl -x "http://customer-acme-country-de-city-berlin-session-a91:$PROXY_KEY@proxy.your-provider.com:8080" \
  -H "Accept-Language: de-DE,de;q=0.9" \
  https://www.example.de/product/42

const PROFILES = {
  de: { locale: "de-DE", tz: "Europe/Berlin", lang: "de-DE,de;q=0.9", city: "berlin" },
  us: { locale: "en-US", tz: "America/Chicago", lang: "en-US,en;q=0.9", city: "chicago" },
  jp: { locale: "ja-JP", tz: "Asia/Tokyo", lang: "ja-JP,ja;q=0.9", city: "tokyo" },
};

function contextFor(country, sessionId) {
  const p = PROFILES[country];
  return {
    locale: p.locale,
    timezoneId: p.tz,
    extraHTTPHeaders: { "Accept-Language": p.lang },
    proxy: {
      server: "http://proxy.your-provider.com:8080",
      username: `customer-acme-country-${country}-city-${p.city}-session-${sessionId}`,
      password: process.env.PROXY_KEY,
    },
  };
}

How pools get burned

Residential IP quality is a shared, depletable resource. When a pool degrades it degrades for everyone on it, and the behaviours that cause it are all things that feel productive in the moment.

• Hammering one target from a narrow subnet, so the site blocks the /24 rather than the address.
• Ignoring 429s and retrying immediately, which converts a soft rate limit into a durable block.
• Reusing an IP after it has been challenged — the challenge was the signal; using it again confirms the suspicion.
• Perfectly regular request intervals. Real traffic is bursty and irregular; a metronome is not.
• Unbounded concurrency per domain, which is how you take a site down and end up in an abuse report.

Backoff and per-domain concurrency, concretely

Exponential backoff with jitter, a hard cap on attempts, and a concurrency limit that is per-domain rather than global. The per-domain part is the one that gets skipped: a global limit of 100 is fine spread across 50 sites and catastrophic pointed at one.

Retire, don't retry. If a session got challenged, that identity is spent — throw it away and start a new journey rather than trying the same address again with more hope.

import asyncio, random, time
from collections import defaultdict

DOMAIN_LIMITS = defaultdict(lambda: asyncio.Semaphore(8))  # per-domain, not global

async def fetch(url: str, attempts: int = 4):
    host = urlparse(url).netloc
    async with DOMAIN_LIMITS[host]:
        for i in range(attempts):
            sid = uuid.uuid4().hex[:12]        # fresh identity every attempt
            r = await get(url, session=sid)

            if r.status_code == 429:
                wait = float(r.headers.get("Retry-After", 2 ** i))
                await asyncio.sleep(wait + random.uniform(0, 1))
                continue
            if r.status_code in (403, 503) or looks_like_challenge(r.text):
                await asyncio.sleep(2 ** i + random.uniform(0, 1.5))
                continue                        # identity retired, not reused
            return r

        raise RuntimeError(f"{host}: exhausted {attempts} attempts")

Measure per domain, or you're flying blind

An aggregate 96% success rate is a comforting number that hides the site at 40% which happens to be the one your customer cares about. Track success, soft-block rate and latency keyed by domain, and alert on the derivative rather than the level — a host that fell from 98% to 91% overnight is telling you something changed, even though 91% still looks acceptable on a dashboard.

Also track cost per useful row, not cost per request. A retry storm that eventually succeeds looks fine in a success-rate chart and terrible in a cost chart, and the cost chart is the honest one — every delivered response is billed, including the blocks and the 404s you retried through.

select
  host,
  count(*)                                             as requests,
  avg((status = 200 and not soft_block)::int)          as success_rate,
  avg(soft_block::int)                                 as soft_block_rate,
  percentile_cont(0.95) within group (order by ms)     as p95_ms,
  sum(credits)::float / nullif(count(*) filter (where useful), 0) as credits_per_row
from fetch_log
where captured_at > now() - interval '24 hours'
group by host
having count(*) > 100
order by success_rate asc
limit 20;

Premium residential vs standard, and when it's worth it

Standard datacenter IPs are fast, cheap and completely fine for permissive targets — documentation sites, open data portals, most APIs, anything without a bot-management vendor in front of it. Reaching for residential everywhere is a common way to overspend by a large multiple.

Residential IPs come from real consumer connections, which is why protected sites treat them as ordinary visitors. Use them where they change the outcome: retail, travel, social, ticketing, anything with regional pricing. The decision rule is empirical — try standard, measure the soft-block rate, and upgrade the hosts where it's bad rather than the whole crawl.

# Standard: cheap, fine for permissive hosts
curl -G https://app.irmu.com/api/crawl \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  --data-urlencode "url=https://docs.example.com/reference" \
  --data-urlencode "js=false"

# Premium residential + city targeting: for protected, geo-sensitive targets
curl -G https://app.irmu.com/api/crawl \
  -H "Authorization: Bearer $IRMU_API_KEY" \
  --data-urlencode "url=https://retail.example.de/product/42" \
  --data-urlencode "premium=true" \
  --data-urlencode "country=de"

The checklist

One IP per logical session. Cookies and IP rotate together. Geography, locale, timezone and language all agree. Exponential backoff with jitter and a hard attempt cap. Concurrency capped per domain. Challenged identities retired, never reused. Metrics per host, alerting on change. Residential only where it changes the outcome.

None of it is clever. All of it is the difference between a crawl that quietly works for a year and one that someone babysits every morning.