Scrape product pages at scale: from a catalogue map to a delivered table

Last quarter I took a catalogue with a shade over 200 000 items. The brief read like every other one: title, price band, availability, image list, category path, seller name, delivered as a table a pricing team could open on Monday morning. The site made it interesting. Three separate doors into the same inventory, none of them agreeing on how many items existed, and a paginated grid that reshuffled itself while I walked it.

That job produced the pipeline I still run. Five moving parts: a map of what exists, a fetch layer that talks to the interface the page itself talks to, a field extractor with fallbacks, a change detector keyed on a row digest, and a state store that lets a broken run resume from the last confirmed position. Everything below is that pipeline, with the arithmetic I use to size it and the checks I run before I sign anything off.

The code is Python. I have trimmed the logging and the argument parsing out of it. The rest runs as written.

Three ways into a catalogue, and how I pick one

Before a single card gets fetched I need to know how many there are and where they live. There are three sources for that, and they disagree with each other on every site I have measured.

The sitemap is the cheapest. A sitemap_index.xml pointing at gzipped shards, each holding up to 50 000 URLs with a lastmod stamp. Forty one shards covered my 200 000 items and cost 41 requests. The catch is what the sitemap represents: the set of pages the site wants indexed, which drops discontinued items, unlisted variants and anything behind a facet the operator considers thin. My count from the sitemap came out at 186 402. Reality was higher.

The category walk is the honest one. Start at the top of the tree, descend into every leaf, page through each grid. It finds items no sitemap mentions. It also costs a request per grid page plus one per card, it double counts anything filed under two branches, and on a deep tree it takes hours before you have a count at all.

The third door is the interface the grid itself calls. Load a category page with the network panel open, filter to fetch and XHR, and the grid usually turns out to be a JSON payload arriving after the HTML. That payload carries a total field, a page size, and the fields the grid renders. My count from there was 214 880 claimed, 200 118 unique once I deduplicated. Both other sources were wrong, in opposite directions.

I use all three, weighted. The sitemap seeds the queue in the first ten seconds so workers have something to do. The interface produces the authoritative enumeration. The category walk runs once a week over the branches where the other two disagree by more than two percent, and it has caught real gaps twice.

The interface the page already calls

Parsing markup is a maintenance contract you sign with someone who never told you the terms. The grid interface is a contract too, and it changes less often, returns more fields, and costs a fraction of the bytes.

Finding it takes ten minutes. Open a category page, clear the network panel, scroll to trigger the next batch, and look for a request whose response is JSON with an array of items in it. Copy it as cURL. Then strip it down: remove one header, replay, see if it still answers. On the catalogue above the minimum turned out to be three headers and two query parameters. Everything else in the browser's version was decoration.

import httpx, json

BASE = "https://shop.example.net/api/v2/catalog/search"
HEAD = {
    "accept": "application/json",
    "x-requested-with": "XMLHttpRequest",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/141.0 Safari/537.36",
}

def list_page(client: httpx.Client, cat: str, cursor: str | None, size: int = 48):
    params = {"category": cat, "limit": size, "sort": "id_asc"}
    if cursor:
        params["after"] = cursor
    r = client.get(BASE, params=params, headers=HEAD, timeout=20.0)
    r.raise_for_status()
    doc = r.json()
    return doc["items"], doc.get("cursor"), doc.get("total")

Three details decide whether this holds up over a long run.

The first is the signature parameter. Some grids sign each call with a token computed in JavaScript from a timestamp and a salt baked into the bundle. When I meet one I read the bundle, port the twenty lines of arithmetic into Python, and pin the salt in a config file with a checker that alerts when the bundle hash changes. Two of my running jobs need this. Both salts have survived four months.

The second is the embedded payload. Plenty of frameworks ship the first page of results inside the HTML as a <script id="__NEXT_DATA__" type="application/json"> block or a window.__STATE__ assignment. That gives me a free page zero on any card fetch and a reference copy of the record shape for validation.

The third is the field surplus. Grid payloads routinely carry things the visible page never renders: internal stock counts, warehouse codes, a canonical parent identifier for variants, a updated_at timestamp. On this catalogue the payload held a revision integer that incremented on every editorial change, which later saved me the entire change detection stage for 60 percent of rows.

Pagination that lies to you

Walking a paginated grid feels like the simple part. It is where I have lost the most data, and the losses are silent.

Three failures show up on almost every large catalogue. Offset pagination over a mutable sort is the first. If the grid sorts by popularity or by "newest" and I page through it over forty minutes, items move between pages while I walk. Something that sat at position 340 slides to 290 while I am fetching page 8, so I see it twice; something at 292 slides to 350 and I never see it at all. On my first pass over this catalogue I collected 197 400 rows and 6 100 of them were duplicates, which means I had actually missed about 8 800 items.

The fix is a stable sort key. Ask the interface for sort=id_asc, and if it refuses, ask for a cursor. Keyset pagination survives mutation because the cursor encodes a position in a total order, so inserts and deletes shift nothing that has already passed.

Depth ceilings are the second. Most search backends refuse an offset beyond some number, and the polite ones return an error while the rude ones return page one with a status 200. Mine capped at offset 10 000, which is 208 pages of 48. My catalogue needs 4 167 pages. So the enumeration has to be sharded until every shard fits under the ceiling.

def shard_plan(counts: dict[str, int], ceiling: int = 10_000, size: int = 48) -> list[dict]:
    """Split any facet whose count exceeds the reachable depth."""
    plan, oversize = [], []
    for facet, n in counts.items():
        if n <= ceiling:
            plan.append({"facet": facet, "pages": -(-n // size)})
        else:
            oversize.append((facet, n))
    for facet, n in oversize:
        parts = -(-n // ceiling) + 1          # one spare part for growth
        for band in price_bands(facet, parts):
            plan.append({"facet": facet, "price": band, "pages": -(-ceiling // size)})
    return plan

I split by category first, then by price band inside any category that stays too large, then by first character of the title as a last resort. Sixty four shards covered the catalogue with the largest at 7 900 items, comfortably under the ceiling with room for the inventory to grow.

The third failure is the total count field. It reported 214 880 while unique identifiers came to 200 118. Nobody at the shop was lying; the counter includes variants that the grid collapses into a single card. I record both numbers and I gate on the unique count, with the ratio between them tracked across runs. When that ratio moves by more than three points I go and look, because it usually means the grid changed how it groups variants.

Fields that survive a markup edit

A field extractor written against one snapshot of a page has a lifespan measured in weeks. Mine is a chain: take the payload value, fall back to a structured data block, fall back to a markup selector, and record which rung answered.

That last part matters more than the fallbacks. Every row carries a src map saying where each field came from, so when a run reports that 12 percent of prices now come from the third rung, I know the payload shape changed before any downstream consumer notices a thing.

from selectolax.parser import HTMLParser
import json, re

def pick(payload: dict, html: str | None, spec: list):
    """spec: list of (kind, expression) tried in order."""
    for rung, (kind, expr) in enumerate(spec):
        try:
            if kind == "json":
                cur = payload
                for k in expr.split("."):
                    cur = cur[int(k)] if k.isdigit() else cur[k]
                if cur not in (None, "", []):
                    return cur, rung
            elif kind == "ld" and html:
                for blk in HTMLParser(html).css('script[type="application/ld+json"]'):
                    doc = json.loads(blk.text())
                    node = doc[0] if isinstance(doc, list) else doc
                    if expr in node and node[expr]:
                        return node[expr], rung
            elif kind == "css" and html:
                node = HTMLParser(html).css_first(expr)
                if node and node.text(strip=True):
                    return node.text(strip=True), rung
        except (KeyError, IndexError, TypeError, ValueError):
            continue
    return None, -1

The specs themselves are short, and I keep them in one file per site.

Two normalisation rules go in before storage, and both exist because of specific damage. Prices become integers in minor units after stripping every character outside the digits and one separator, because a locale switch turned decimal commas into decimal points halfway through a run and my float parser accepted both silently. Image URLs get their query strings removed, because the CDN appends a signature that changes on every request and would otherwise mark every row as modified.

Telling changed cards from untouched ones

The first run collects everything. Every run after that should collect what moved, and knowing what moved is a digest problem.

I hash the stable part of each record. Volatile fields stay out of the digest: view counters, "seven people are looking at this", signed image parameters, any timestamp the site generates per request. What goes in is the tuple a pricing team would call the product.

import hashlib, json

DIGEST_FIELDS = ("title", "price_current", "price_was", "availability",
                 "category_path", "image_urls", "seller", "variant_ids")

def row_digest(row: dict) -> str:
    payload = []
    for f in DIGEST_FIELDS:
        v = row.get(f)
        if isinstance(v, list):
            v = sorted(str(x) for x in v)
        payload.append((f, v))
    blob = json.dumps(payload, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":")).encode("utf-8")
    return hashlib.blake2b(blob, digest_size=16).hexdigest()

Sorting the list fields is the part people leave out, and it produces a run where 40 percent of rows look modified because the interface returned the same three image URLs in a different order.

With a digest column the comparison between two runs becomes one query, and the four outcomes it produces each mean something different. A digest that matches means skip the detail fetch entirely. A digest that differs means write a new version row and record which fields moved. An identifier present last run and absent now is a delisting, and I hold those for three runs before marking them gone, because a single failed shard can make live items look deleted. An identifier that appears for the first time is a new listing, and I check those against the sitemap to confirm the site agrees.

On this catalogue the churn settled at about 7 percent per day, which cut the daily detail fetch from 200 000 requests to roughly 14 000. That reduction is the difference between a run that fits inside a maintenance window and a run that does not. The revision integer in the payload made it better still: where the site increments a revision, I trust it and skip the digest comparison, and that covered 60 percent of rows.

What 200 000 cards actually cost

Sizing a run is arithmetic, and doing it before the run saves the argument afterwards. Here is the full first pass over the catalogue, with a failure allowance folded into each stage.

Three things fall out of that table. The detail fetch is 96 percent of the work, so any field I can pull from the grid payload removes 200 000 requests from the plan. Bytes reach 3.38 GB on the first pass and around 580 MB a day after that, which is why I run this on an unmetered pool package and stop thinking about volume as a variable at all. And the retry allowance is 4 percent for a reason: that is the failure share I measure on a well behaved origin at a sane pace, and a run planned at zero percent overruns its window on the first bad hour.

Wall clock depends entirely on the pace I hold. Same request count, four different postures:

The bottom row is the trap. It looks like the fastest option until the retry queue starts feeding itself, at which point effective throughput drops below the 80 per second row and the origin has a very clear picture of what I am doing. I run at 40 per second on this catalogue and finish inside the window with margin. Before committing to a pace I work the concurrency figure out on paper first, because it has to fit the package ceiling as well as the origin's tolerance.

Spreading the run across addresses and holding a pace

Aggregate pace is one number. What the origin sees is per address, and that is the number that decides whether the run finishes.

The pool I work from holds around 12 000 active addresses with rotation handled automatically inside it, so my job never picks an exit and never needs to. What my side controls is concurrency and timing. At 160 concurrent tunnels running 15 requests per minute each, any single exit the origin observes is producing well under one request per second, which sits below the threshold on every catalogue I have measured. The exit addresses cycling on the service side do the spreading; my pacing keeps each observed exit quiet.

Thread arithmetic comes from the package. A regular tier gives 1 000 concurrent connections and the corporate tier goes to 3 000. Packages do not stack, and binding a second source address splits the count in half, so with two bound addresses on a regular tier I have 500 to spend across everything running. My collector takes 160, the health checker beside it takes 40, and the rest stays free for the ad hoc work that always appears mid run.

import asyncio, random, time

class Pacer:
    """Token bucket at the aggregate level, jitter at the request level."""
    def __init__(self, rate: float, burst: int = 20):
        self.rate, self.burst = rate, burst
        self.tokens, self.ts = float(burst), time.monotonic()
        self.lock = asyncio.Lock()

    async def take(self):
        async with self.lock:
            now = time.monotonic()
            self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate)
            self.ts = now
            if self.tokens < 1:
                await asyncio.sleep((1 - self.tokens) / self.rate)
                self.tokens = 0.0
            else:
                self.tokens -= 1
        await asyncio.sleep(random.uniform(0.0, 0.35))

The jitter at the end costs almost nothing and removes the metronome signature that a fixed sleep produces. I also hold every worker to max_connections=2 on its client, because a pooled transport left to its own devices keeps sockets alive long after a request returns and quietly consumes the thread budget the semaphore believes it is protecting.

When a shard starts returning 429 I halve its pace, wait 90 seconds, and step it back up by 20 percent every clear minute. That control loop lives per shard, so a strict category slows down while the rest of the run continues at full speed. Running the whole job over addresses aimed at collection work keeps the latency distribution narrow enough that these thresholds stay meaningful from one run to the next.

Resuming a broken run from stored state

Any run measured in hours will be interrupted. A laptop sleeps, a container gets rescheduled, the origin has a bad ten minutes. The pipeline treats interruption as normal and stores enough state to make resumption cheap.

Two tables carry it. One holds the shard plan with a cursor and a status per shard. The other holds the rows, keyed on the product identifier, with the digest and the run identifier that last touched it.

CREATE TABLE IF NOT EXISTS shards (
  shard_id   TEXT PRIMARY KEY,
  facet      TEXT NOT NULL,
  price_band TEXT,
  cursor     TEXT,
  pages_done INTEGER NOT NULL DEFAULT 0,
  pages_plan INTEGER NOT NULL,
  status     TEXT NOT NULL DEFAULT 'pending',
  leased_at  INTEGER,
  attempts   INTEGER NOT NULL DEFAULT 0,
  note       TEXT
);

CREATE TABLE IF NOT EXISTS cards (
  product_id TEXT PRIMARY KEY,
  digest     TEXT NOT NULL,
  payload    TEXT NOT NULL,
  src_map    TEXT NOT NULL,
  first_seen INTEGER NOT NULL,
  last_seen  INTEGER NOT NULL,
  run_id     INTEGER NOT NULL
);

CREATE INDEX IF NOT EXISTS cards_run  ON cards(run_id);
CREATE INDEX IF NOT EXISTS shards_st  ON shards(status, leased_at);

The cursor column is what makes resumption cheap. A shard that died at page 130 of 208 restarts at page 130, because the cursor was written after every successful page inside the same transaction as the rows it produced. Writes are idempotent: INSERT ... ON CONFLICT(product_id) DO UPDATE with last_seen refreshed and digest compared, so re-fetching a page costs a little time and corrupts nothing.

LEASE = 900  # seconds a worker may hold a shard before it is reclaimed

CLAIM = """
UPDATE shards SET status='running', leased_at=:now, attempts=attempts+1
WHERE shard_id = (
  SELECT shard_id FROM shards
  WHERE status='pending'
     OR (status='running' AND leased_at < :now - :lease)
  ORDER BY attempts ASC, pages_plan DESC LIMIT 1
)
RETURNING shard_id, facet, price_band, cursor, pages_done
"""

The lease is the piece I added after losing a night's work. Without it a worker that dies while holding a shard leaves that shard marked running forever, and the run reports success with a hole in it. With a lease, any shard whose worker went quiet for 15 minutes returns to the queue automatically and gets picked up by whoever is free. Two extra guards run beside it: SQLite in WAL mode so readers never block the writer, and a run summary row written at the end recording the plan, the completed shard count and the elapsed seconds, so a run that finished with 63 of 64 shards can never be mistaken for a full one.

Signing off on the output

A table is finished when it passes acceptance, and acceptance is a set of queries with numbers attached. I run them before anyone else sees the file.

Two of these have saved me from shipping bad data more than once. The extractor rung mix caught a payload rename three hours after the site deployed it, while every row still looked populated because the markup fallback was answering. The churn band caught a run where the price field had switched separators, marking 71 percent of rows as modified when almost nothing had moved.

The last piece of sign off is boring and I do it anyway: open the file, sort by price ascending, read the first twenty rows, then sort descending and read twenty more. Extremes are where parsing damage collects. Thirty seconds of reading has caught things no query of mine was written to look for.

Stability across the whole run comes back to what the fetch layer sits on. I keep this pipeline on private server side endpoints so the latency distribution stays predictable enough for the pacing arithmetic to hold, take the list as IP:PORT:LOGIN:PASS from the panel at the start of each run, and size the term to the job with IPv4 endpoints sold by term when a catalogue needs a month of daily passes. A short trial window is enough to run one sharded enumeration end to end and see the real numbers for your own origin. Since volume stops mattering on term based access to the pool, the only variable left to tune is pace, and pace is the one I control.

Three companion pieces cover what this one skipped. The full flow from collection to alerting lives in my write up on building a price monitoring pipeline, which picks up where this table gets delivered. Listings with a different shape, heavy pagination and thin records, are worked through in collecting business directory records. And for postings that expire while you walk them, gathering job board listings covers the freshness handling that a product catalogue never needs. Before you fix the pace for a run of this size, the pool and thread calculator turns the request count and the package ceiling into the concurrency number to hold.