403, 429, and a Wall of CAPTCHAs: The Emergency Room Guide for Blocked Scrapers

Every scraping pipeline eventually hits the wall — a 403 where yesterday’s requests succeeded, a 429 that no amount of sleeping fixes, a CAPTCHA page where product data used to be. The emergency response is usually the wrong one: more retries, more threads, more aggression. This is the ER guide — triage by error code, the treatments that actually work, and the architectural admission that ends the recurring emergency.

Triage: Read the Error First

Blocked scrapers get treated with folklore. The actual practice is triage — the error code is a diagnosis, and the diagnoses differ:

The meta-lesson in that table: the 200-with-stale-data row is the dangerous one. Every other symptom announces itself; a cached, plausible page passes silently into your database wearing yesterday’s prices.

Treatment Path One: The Credibility Prescription

The core medicine for 403-class blocks isn’t volume — it’s traffic that looks like it belongs. Concretely, the properties that move block rates:

• Residential exit IPs, geo-matched to the target’s market (a datacenter IP in a different country announces itself)
• Session coherence — sticky sessions for multi-step flows (a session that changes IP mid-funnel looks like exactly what it is), with rotation across separate tasks
• Browser-grade rendering where the target’s page requires it — JS rendering and fingerprint handling, not just header spoofing
• Polite pacing per session, distributed across many sessions rather than fast on one

Thordata’s stack prices each property at published rates: residential traffic on the volume slider ($2.00/GB at 1 GB down to $0.65/GB at scale), country/state/city/ASN targeting at no extra charge, sticky sessions up to 90 minutes, and a Scraping Browser tier at $2.50/GB for the flows that need full browser automation.

Treatment Path Two: The Delegation Surgery

When the blocking is structural — the target invests in anti-bot because its data is valuable — the ER question changes from “how do we get through?” to “should getting through be our job at all?” The surgical option is delegation: an unlocker tier that handles IP rotation, fingerprinting, JS rendering, and CAPTCHA solving upstream of your bill. Thordata’s Web Unlocker runs roughly $1.00–$1.30 per 1,000 responses, and for the standard-target version of the same surgery, the Web Scraper API (120+ maintained targets at about $0.50–$1.00 per 1,000 delivered results) takes the whole parser-maintenance problem with it.

The economic logic of the surgery is the part worth internalizing: every failed attempt on your own infrastructure burns budget while producing nothing. Managed tiers that bill on *delivery* convert your block-rate risk into the vendor’s problem — and a vendor with a 100M+ residential pool across 190+ countries amortizes that risk across every customer they have.

The Architectural Admission

The ER visit that repeats weekly is a signal, not bad luck. The admission worth making:

# The pattern that keeps pipelines out of the ER
def fetch(url, market):
    try:
        return scraper_api.get(url, geo=market)   # managed, billed on delivery
    except TargetNotCovered:
        return unlocker.get(url, geo=market)      # the stubborn middle tier
    # raw residential only for the genuinely simple long tail

Three tiers, one routing rule: managed scrapers for standard targets, unlocker for the hostile middle, raw residential for the simple tail. The emergency stops being an emergency when the architecture expects blocks and routes around them by design.

FAQ

We got blocked “overnight” — nothing changed on our side. What changed?
The target’s. Anti-bot systems ratchet up thresholds on schedule (and especially ahead of high-value traffic periods like sales events), so yesterday’s polite rate becomes today’s pattern. The monitoring discipline that catches this: track block rate per target per day, so the overnight change shows as a step on a chart instead of a surprise in a pager.

Is solving CAPTCHAs ourselves ever the right move?
As a hobby, sure; as an architecture, rarely. CAPTCHA solving is a specialized arms race with its own vendors and its own quality curve — managed layers bundle it upstream, and the per-response pricing of an unlocker tier is usually cheaper than your engineer-hours spent gluing solvers into retry loops. Solving costs should be invisible to your bill, which is what “billed on delivery” means in practice.

What about rotating proxies we already have?
Rotation is one property of credible traffic — the table’s 403 row lists the others (geo-match, session coherence, rendering). A rotation-only fix on a fingerprint-detected target produces the same 403 from more addresses. Test the full prescription before concluding the medicine doesn’t work.

How does this map to search-engine targets specifically?
It’s the most acute case: SERPs are the best-defended pages on the web, and the “scrape google without getting blocked” recipe list online is a graveyard of last-year’s tricks. The modern answer is to stop scraping and buy structured responses: per-response SERP APIs at roughly $0.70–$1.20 per 1,000, or — for agent workloads — an MCP endpoint the agent drives itself; the Thordata MCP server wraps live multi-engine search, request history, and usage statistics as agent-discoverable tools, with new accounts getting 500 free calls to test the whole prescription.

When is being blocked the correct outcome?
When the data isn’t public or the target’s terms exclude you — a login-walled internals page returning 403 is the system working. The ER guide covers public-data collection; the parallel legal review covers which of your targets this guide applies to. Responsible scraping starts with that list, not with the retry loop.

The discharge instructions?
Monitor block rates per target, route by target difficulty (managed / unlocker / raw), verify observation freshness so cached 200s don’t masquerade as health, and put search targets on structured per-response delivery from day one. The pipelines that follow the instructions stop visiting the ER — and the agent-driven version of the same delivery never learns the folklore in the first place.