Scrape business directories: from the rubric tree to one folded table

A regional directory reached me with a plain brief. Every organisation in the city as one row, with phones, hours and an activity label, delivered as a table the sales floor could work through the same week. The site claimed 80 000 cards. My first pass came back with 63 400 rows, and once I looked properly, a quarter of them were the same businesses filed twice under different activity labels.

Two more passes fixed that, and what came out of them is the pipeline I now point at any directory of this shape. Read the rubric tree. Size every branch against the depth the pager will actually serve. Cut the oversized branches until each part is reachable. Walk the listings for identifiers, fetch cards only for identifiers I have never seen, fold the duplicates on a stable key, then compare the run against the previous one to find who closed and who opened.

The code is Python. I have taken the logging and the argument parsing out of the snippets below; the rest is running as written.

What a directory looks like under the pages

Three layers sit under every business directory I have taken apart, and the shape barely varies from one to the next.

The rubricator is the top layer. A tree of activity labels, usually three levels deep: a group such as automotive services, a rubric such as tyre fitting, sometimes a narrower leaf below that. My directory held 48 groups and 612 rubrics. A second axis crosses the tree, and that axis is geography: 24 districts, with metro stations and street segments underneath them. The cross product of activity and geography is the real address space of the catalogue, and it runs into the hundreds of thousands of combinations while the card count sits at 80 000.

The listing page is the middle layer. Twenty short cards per page on mine, each carrying the numeric identifier, the display name, a rating, one address line and a truncated rubric list. Phone numbers are absent from listings on almost every directory I have measured, and that absence is a deliberate piece of design. Contacts are the product these sites sell to their own advertisers, so they sit one click deeper where the site can count the reveal.

The organisation card is the bottom layer and the only place the full record lives. Display name, legal name where the directory holds one, the complete rubric list, coordinates, opening hours per weekday, phone numbers, website, email, photo count, review count, and a status flag that moves to temporarily closed or permanently closed. On my target the card also fired a second call after render to fetch contacts, which turns one page into two requests for the rows that matter most.

Why I walk the rubric tree and leave the search box alone

Search looks like the shortcut. Type a rubric name, read the results, page through, repeat 612 times. I measured both routes on the same directory over one weekend and the gap between them decided the whole project.

Search ranks before it returns. That means a cap, and mine capped at 500 results per query, which is 25 pages of 20. Any rubric holding more than 500 organisations gets silently truncated to the top slice by whatever the ranker considers quality. Search also applies a filter of its own: cards with no photo, no reviews and no verified phone were absent from results while sitting happily in the rubric listing. And chains came back collapsed, one entry standing in for 40 branches, with the branch list hidden behind an expander that the result payload did not carry.

The rubric walk has none of that. The tree is the site's own partition of its own data. Every card carries at least one rubric, because a card with no rubric cannot be filed anywhere and the moderation queue rejects it. So covering every rubric covers every card by construction, and I can prove the coverage arithmetically before a single row is delivered.

The numbers were not close. Search across all 612 rubric names produced 41 900 distinct identifiers over 9 100 requests. The rubric walk produced 80 118 distinct identifiers over 7 881 listing pages. Fewer requests, nearly double the coverage. I keep the search endpoint in the toolbox for one job only: spot checks against a handful of names I expect to see, run after the walk finishes, as a sanity probe on branches that came back thinner than the previous week.

The depth ceiling, and cutting a rubric until it fits

Every listing has a floor below which it stops serving, and the polite sites tell you. Mine did not. The pager accepted page 50, and for page 51 and everything beyond it returned page 50 again with status 200 and no warning of any kind. A walker that trusts the pager will loop on those twenty identifiers until the run ends.

So the first thing every part of my plan carries is a reachability figure. Fifty pages of twenty puts the ceiling at 1 000 positions. Of my 612 rubrics, 96 held more than 1 000 cards, and the largest, beauty and grooming, held 6 480. Those 96 rubrics covered 61 percent of the catalogue, which means the naive walk would have missed most of the city.

Cutting is recursive along the axes the site already offers. District first, since 24 districts split almost every oversized rubric below the ceiling in one step. First letter band second, for the handful of rubrics that stay too large inside a single dense district. Rubric leaf third, where the tree has one. After cutting, my 612 rubrics became 1 340 parts, and the largest part held 940 cards with room to grow before it needs another cut.

CEILING, PAGE = 1000, 20
AXES = ("district", "letter", "leaf")

def plan_parts(rubric, count, probe, axis=0, filters=None):
    """Cut a rubric along the next axis until every part is reachable."""
    filters = filters or {}
    if count <= CEILING or axis >= len(AXES):
        return [{
            "rubric": rubric,
            "filters": filters,
            "pages": min(-(-count // PAGE), CEILING // PAGE),
            "truncated": count > CEILING,
        }]
    parts = []
    for value, sub in probe(rubric, filters, AXES[axis]).items():
        if sub:
            parts += plan_parts(rubric, sub, probe,
                                axis + 1, {**filters, AXES[axis]: value})
    return parts

The probe callable asks the directory for a count under a filter combination without pulling any cards. On my target the count came back in the listing payload as a found integer on page one, so a probe costs one request and returns the number that drives the plan. Six hundred and sixty probe requests bought me a plan I could size, schedule and check afterwards.

Detecting the silent ceiling during a run is worth the eight lines it takes. I hash the sorted identifier set of each listing page and compare it with the previous page. Two consecutive pages carrying an identical hash means the pager has stopped moving, so the worker marks the part as truncated, writes the reason and hands it back to the planner for a further cut. That check has fired on three separate directories where my count probe reported a number the pager refused to serve.

Fields on an organisation card and how long each one holds

A field spec written against one snapshot of a card has a working life measured in weeks. Mine is a chain of sources per field, and every extracted value records which link in the chain answered, so a shift in the payload shows up as a change in the source mix long before anyone downstream notices bad data.

Two normalisation rules run before anything reaches storage. Phones become digit strings with the country prefix stripped and the last ten digits kept, because the same number arrives formatted five different ways across one directory and a raw string comparison folds nothing. Opening hours become seven pairs of minutes past midnight, with a flag for round the clock and a flag for closed, because the free text version defeated every downstream filter the sales team tried to apply to it.

The source mix is the number I watch across runs. When phones start arriving from the fallback link on more than five percent of cards, the contacts call has changed shape and I have hours to fix it before the daily table goes out with holes in the column that matters most.

Folding one organisation out of five rubrics

An organisation sits in 1.8 rubrics on average, and the busy ones sit in six. That produces 144 212 listing rows for 80 118 identifiers, and folding those is trivial: the identifier is the same in every rubric, so a set does the work.

The interesting duplicates carry different identifiers. A business submitted twice by two different people. A branch page and a head office page describing the same address. A card created by the moderation team and a second one claimed later by the owner. On my directory 2 640 organisations existed under more than one identifier, and no amount of paging discipline would have found them, because the site itself considers them separate records.

Folding those needs identity keys computed from the content of the card, then a merge across whichever keys agree.

import re, unicodedata

LEGAL = {"ltd", "llc", "inc", "gmbh", "plc", "co", "company", "group", "holdings"}

def norm_name(s: str) -> frozenset:
    s = unicodedata.normalize("NFKD", s or "").casefold()
    s = re.sub(r"[^\w\s]", " ", s)
    return frozenset(t for t in s.split() if len(t) > 1 and t not in LEGAL)

def phone_digits(raw: str):
    d = re.sub(r"\D", "", raw or "")
    return d[-10:] if len(d) >= 10 else None

def geo_cell(lat, lon, step=0.0009):          # roughly 100 metres per side
    return int(lat / step), int(lon / step)

def identity_keys(card: dict):
    for p in card.get("phones", []):
        d = phone_digits(p)
        if d:
            yield ("tel", d)
    host = registrable(card.get("site"))
    if host:
        yield ("web", host)
    if card.get("lat") and card.get("geo_precision") != "city":
        yield ("geo", geo_cell(card["lat"], card["lon"]), norm_name(card["name"]))

The keys go into buckets, and every bucket joins its members into one cluster. A disjoint set structure keeps that cheap over 80 000 cards, and the whole fold runs in under two seconds on my laptop.

class Union:
    def __init__(self):
        self.p = {}

    def find(self, x):
        self.p.setdefault(x, x)
        while self.p[x] != x:
            self.p[x] = self.p[self.p[x]]
            x = self.p[x]
        return x

    def join(self, a, b):
        ra, rb = self.find(a), self.find(b)
        if ra != rb:
            self.p[rb] = ra

def fold(cards, cap=4):
    buckets = {}
    for c in cards:
        for k in identity_keys(c):
            buckets.setdefault(k, set()).add(c["org_id"])
    u = Union()
    for key, ids in buckets.items():
        if len(ids) > cap:        # hotline, franchise domain, default map point
            continue
        ids = sorted(ids)
        for other in ids[1:]:
            u.join(ids[0], other)
    return {c["org_id"]: u.find(c["org_id"]) for c in cards}

Picking a survivor inside each cluster follows fixed precedence: the card with a legal name wins, then the card with the most filled fields, then the oldest identifier. Everything the survivor lacks gets filled from its siblings, and the sibling identifiers stay in an aliases column so a later run recognises the cluster immediately without recomputing anything.

Chains, hotlines and the keys that merge too much

That cap=4 line hides the whole story of my second failed pass. Phone numbers are a strong identity signal right up to the moment a chain publishes one hotline across every branch it owns.

A supermarket chain on my directory listed 220 branches, all carrying the same eight digit hotline. The phone key merged all 220 into a single organisation, and because the survivor rule picked the card with the most filled fields, the table went out with one supermarket where there should have been 220 rows with 220 addresses. Nobody on the sales floor noticed for two days. I noticed when the district totals stopped adding up against the count probes I had stored at plan time.

The cap fixed it and cost almost nothing. Any key value that appears on more than four cards is dropped from the merge entirely, on the reasoning that a genuine duplicate pair or triple is common while a five way match on one signal means the signal is shared infrastructure. On my run 812 phone values exceeded the cap and covered 11 300 cards, all of them chains and call centres.

The same treatment goes to two other over merging keys. Franchise websites point every branch at one domain, so the web key gets a cap of three. And 1 940 cards carried coordinates at the exact geographic centre of the city, which is what the directory writes when geocoding an address fails. Those cards get geo_precision set to city in my normaliser and drop out of the geo key completely, since a hundred metre cell containing 1 940 organisations merges nothing useful.

One case stays deliberately unresolved. Two branches of the same chain trading inside one shopping centre share a building, a name and sometimes a floor. The geo key merges them and I let it, because the directory itself cannot tell them apart either and the sales team dials the same number for both.

Reading closures and openings between two runs

A directory table that is refreshed weekly earns its value from the delta. Which organisations opened, which closed, which moved. Getting that right needs more care than comparing two identifier sets, because a run with a failed part looks exactly like a district that went out of business.

Three signals point at a closure and they carry different weight. The card returning status 410 is definitive, since the directory has retired the record. The card carrying a permanently_closed state is definitive as well and arrives earlier, because moderators set the flag before the record is retired. The third signal, an identifier that stops appearing in any listing, is the weak one, and it is the one that produced 2 100 phantom closures on the run where a district filter silently returned empty for three rubric parts.

The hold down rule solved that. An identifier missing from the listings gets a miss counter, and only two consecutive misses write a closure. Definitive signals skip the counter. Moves are separate: the identity keys survive, the address key changes, and those rows go to the sales team as updates with the previous address attached.

HOLD = 2

def reconcile(cur: dict, prev: dict, misses: dict):
    opened, closed, moved = [], [], []
    for oid, row in cur.items():
        old = prev.get(oid)
        if old is None:
            opened.append(oid)
        elif row["addr_key"] != old["addr_key"]:
            moved.append(oid)
        misses[oid] = 0
    for oid in prev.keys() - cur.keys():
        misses[oid] = misses.get(oid, 0) + 1
        hard = prev[oid]["status"] in ("gone", "permanently_closed")
        if hard or misses[oid] >= HOLD:
            closed.append(oid)
    return opened, closed, moved

Openings need their own guard. An identifier appearing for the first time should be reachable from at least one listing page, and a card that answers on a direct URL while showing up in no rubric at all is usually a draft the owner has not finished. I hold those in a pending table and promote them when a listing confirms them. Weekly numbers on my directory settled at 1 180 openings, 640 closures and roughly 300 moves, against a card base of 80 000, which is close to two percent turnover a week.

What a walk over 80 000 cards costs

Sizing the run before it starts turns an argument about scheduling into arithmetic. Here is the full first pass, with a failure allowance folded into every stage.

Three readings come out of that table. Card detail is 75 percent of the requests and 81 percent of the bytes, so every field I can pull from the listing payload takes real weight off the plan. The contacts call is small in bytes and large in count, and it is the stage the directory watches hardest, since reveals are what it charges its advertisers for. And the weekly refresh is a sixth of the first pass, which means the expensive part happens once and the schedule after that is comfortable.

Volume stops being a planning variable when access is sold by term, so I run this on unmetered access to the pool and put the whole two gigabytes through without arithmetic about bytes. What stays variable is wall clock, and wall clock is a function of the pace I hold.

I hold 30 per second on this directory. The bottom row is where a run goes wrong: retries start generating retries, effective throughput sinks below the 60 per second row, and the directory now has a very legible picture of the traffic it is serving.

Pace, address spread and the package ceiling

Aggregate pace is one number and the directory never sees it. What it sees is per exit, and that number decides whether the walk finishes.

The pool I work from carries around 12 000 active addresses with rotation handled automatically inside it, so my code never picks an exit and never tracks one. At 120 concurrent tunnels doing 15 requests a minute each, any single exit the directory observes is producing a quarter of a request per second, which sits far below the trigger on every catalogue of this size I have measured. A rotating pool of server endpoints does the spreading; my pacing keeps every observed exit quiet enough that the spreading never has to save me.

Thread arithmetic comes from the package. Regular tiers allow 1 000 parallel connections and the corporate tier goes to 3 000. Two packages held at once do not add up, and two bound source addresses divide the ceiling between them, so with both slots in use on a regular tier I have 500 threads to spend across everything running on that machine. The directory walker takes 120, the checker beside it takes 30, and I keep the rest free for the unplanned work that always turns up in the middle of a long run.

import asyncio, random

class PartRate:
    """Per part pace: additive rise while quiet, multiplicative cut on refusal."""
    def __init__(self, start=4.0, floor=0.5, top=12.0):
        self.rate, self.floor, self.top = start, floor, top
        self.gap = 1.0 / start
        self.lock = asyncio.Lock()

    def observe(self, status: int):
        if status in (429, 503):
            self.rate = max(self.floor, self.rate * 0.5)
        elif status == 200:
            self.rate = min(self.top, self.rate + 0.05)
        self.gap = 1.0 / self.rate

    async def wait(self):
        async with self.lock:
            await asyncio.sleep(self.gap * random.uniform(0.7, 1.3))

Holding the controller per part matters more than the numbers inside it. A dense rubric in the central district will start refusing while 1 300 other parts run untouched, and a global controller would drag the whole walk down to the pace of its worst branch. With one controller per part, that branch slows to half a request a second, works through its 47 pages, and the run finishes on schedule.

Two smaller habits go with it. Every client gets max_connections=2, because a pooled transport left alone keeps sockets open long after a response returns and quietly spends the thread budget the semaphore believes it is guarding. And the jitter multiplier on the sleep costs nothing while removing the metronome signature that a fixed interval writes into the directory's access log. Running the walk over the same endpoints my Key Collector batches use keeps the latency spread narrow, which is what lets these thresholds carry over from one week to the next.

Signing the table off, and the holes that stay in it

A directory table is finished when it passes acceptance, and acceptance is a set of queries with numbers attached to them. I run all of these before the file leaves my machine.

Some holes survive every check and I report them as known limits with the table. Hours are self reported and stale on roughly a third of cards, because nobody updates a directory listing after moving to winter opening. Rubrics are chosen by the owner during registration, so a print shop that also sells stationery will file itself under whichever label it thinks brings more calls. Call tracking numbers rotate per session on the advertised cards, which makes the phone column churn on rows where nothing has actually moved; I detect those by fetching the contacts call twice for a two percent sample and flagging any card whose number changed between the two.

The last piece of sign off is manual and takes four minutes. I sort by review count descending, read the top twenty rows, then sort by name and read twenty from the middle of the alphabet. Damage collects at the extremes and in the boring middle for different reasons, and reading forty rows has caught things no query of mine was written to look for.

Keeping all of this repeatable comes back to the layer the fetcher sits on. I run the walk over private server proxies so response times stay predictable enough for the pacing arithmetic to survive from week to week, pull the list as IP:PORT:LOGIN:PASS from the panel at the start of every run, and match the access term to the project with IPv4 endpoints by access term when a directory needs a full quarter of weekly refreshes. A short trial window is long enough to walk one district end to end and see the real refusal rate on your own target. With rotation happening inside the pool, the only lever left on my side is pace, and pace is the one I want to control.

Three companion pieces pick up what this one left out. Catalogues where the record is a product and the pagination is the enemy are worked through in collecting product pages at scale. Listings that expire while you walk them, with all the freshness handling a directory never needs, are covered in gathering job board postings. The full flow from collection through storage to alerting lives in building a price monitoring pipeline. And before fixing the pace for a walk this size, the pool and thread calculator turns your request count and your package ceiling into the concurrency figure to hold.