You Priced the Scraper at $0: Fixing the Most Expensive Number in Data Sourcing

Every build-vs-buy comparison for web data contains a line that says “scraper: $0 in licensing” — and that line quietly decides the whole comparison, wrongly. The scraper’s cost was never zero; it was prepaid in engineering, deferred in maintenance, and compounding in risk. This is a repair manual for that comparison: the five broken lines teams write, what each should say, and what the honest arithmetic looks like once it’s fixed.

The Comparison Everyone Writes

A team needs, say, five million product records across twenty marketplaces. The spreadsheet has two columns: “build” and “buy,” and the build column starts with a number that looks like this:

The decision follows the spreadsheet, the quarter follows the decision, and eighteen months later a retrospective asks why collection eats two engineers and still misses three of the twenty targets. The spreadsheet was the problem. Here’s the repair, line by line.

Repair 1: The $0 License Line Becomes an Engineering Line

The scraper costs what it costs to write — and rewrite. Targets change layouts, deploy anti-bot layers, and shift payloads; a twenty-target collection layer is a permanent maintenance surface, not a one-time build. The honest version of the line: initial build (weeks of engineering at your loaded rate) plus permanent maintenance (measured in the industry at a meaningful fraction of the original build, forever). When a vendor’s Web Scraper API bills per delivered result — Thordata’s runs about $0.50–$1.00 per 1,000 results across 120+ maintained targets, with parser upkeep on the vendor’s side of the ledger — the comparison is not license versus nothing. It’s your maintenance queue versus their maintenance queue, priced per result.

Repair 2: Infrastructure Becomes Failure Economics

“~$80/mo of servers” is the second broken line, because it prices attempts, not deliveries. Blocked requests burn bandwidth without producing records; retries multiply the burn. Per-result pricing is the structural fix — failed attempts cost $0, so your retry strategy stops being a budget negotiation. The same inversion applies one layer down: raw residential traffic for the long tail of targets bills by the gigabyte on a published slider ($2.00/GB at 1 GB down to $0.65/GB at volume), so even the build-your-own path’s raw-materials cost is knowable in advance instead of discovered at month-end.

Repair 3: Time-to-First-Record Gets a Row

The build column’s hidden cost is calendar: weeks or months before the first complete dataset exists. The buy column’s answer is delivery formats and sample-first evaluation — structured records in JSON, CSV, or Parquet, with sample fields viewable before purchase and dataset delivery to S3 as a standard option. Standard catalog datasets start around $0.25 per 1,000 records; custom collection is quoted by scope against the same delivery machinery. When the business need is “this quarter,” the calendar line often decides the comparison before the money lines do.

Repair 4: Risk Gets a Number, Not a Shrug

The build path carries two risks the spreadsheet never prices: collection risk (targets escalate defenses; your pipeline’s coverage quietly erodes) and provenance risk (when a customer or acquirer asks where the training data came from, “we scraped it” is a legally interesting answer). Dataset vendors attach license scope and sourcing lineage to records — per-record fields your legal review can actually review. That’s not compliance theater; it’s the difference between a five-minute answer and a discovery process, and it belongs on the spreadsheet as a row with a real number your counsel will help you write.

Repair 5: The Exit Gets Priced

Build paths lock you in gently: the scraper embodies months of accumulated workarounds, and no vendor will buy it from you. Vendor paths should be checked for the opposite risk — data portability (standard formats, exportable, yours after the contract) and collection portability (can you mirror traffic to another provider? standard proxy endpoints make that a configuration exercise, not a project). Buy, but buy exit-friendly.

The Repaired Spreadsheet

Run your own volume through both columns — a five-million-record need prices out around $2,500–5,000 at catalog per-record rates, or lands in Scraper-API territory if freshness matters more than history. The point of the repair isn’t that buy always wins; it’s that the comparison deserves honest numbers, and the honest numbers usually aren’t close.

FAQ

When does build genuinely win?
When the target is genuinely private or interactive in ways no catalog or API covers — your logged-in workflow, your internal tools. The moment the need is “records of the public web at scale,” the economics flip, and the file you’re reading exists because they flip so consistently. For search-driven needs specifically, the flip is at its most extreme — which is why every serious evaluation in that category ends at a per-response price list or the managed SERP monitoring solution, not at a crawler design doc.

What about scraping APIs as a middle path?
That’s the pragmatic hybrid most teams land on: pre-built scrapers for standard targets (per-result), raw residential for the long tail (per-GB slider), and licensed datasets for history and bulk. One vendor covering all three — as Thordata’s stack does — keeps the layers in one schema and one invoice.

Is there a version of this comparison for search data?
Yes, and it’s the starkest one: collecting SERPs yourself means fighting the most defended pages on the internet, per gigabyte. Per-response pricing — Thordata’s SERP API at roughly $0.70–$1.20 per 1,000 structured responses, or the managed SERP monitoring solution for scheduled tracking — is what that workload should cost. Before you rebuild the spreadsheet for any search-driven product, run a week of continuous SERP data crawling on free trial credits and put the invoice next to your build estimate.

How do we pilot this without a procurement cycle?
Sample-first: pull sample fields for the catalog targets you care about, price your exact volume against published rates, and run one mirrored slice through the Scraper API for a week. The whole evaluation fits in free trial credits and one afternoon of engineering — which is, not coincidentally, the time-to-first-record row of the repaired spreadsheet, demonstrated.

The manual’s one-line summary: the $0 line was never $0, and every comparison built on it inherits the error — fix the five lines, and the decision usually makes itself.