Scrape job boards: depth ceilings, repost duplicates and a snapshot that holds every morning

I keep a daily picture of one national job aggregator for a labour analytics group. The open set sits around 412 000 postings on a normal weekday. Roughly 26 000 appear in a day, roughly 24 000 stop taking applications, and most of the engineering in this job goes into telling a genuine closure apart from a query window that quietly failed to enumerate.

Everything below concerns pages a browser reaches without an account: the search results, the posting pages those results link to, and the category feeds the board publishes so aggregators can pull from it. No login, no application form, no candidate records of any kind. The board's own front end fetches its results from a public endpoint, and I call that same endpoint with the same public parameters the page sends.

Python throughout. Logging and argument handling are stripped from every snippet below, and what remains is what my collector executes.

Three surfaces a board publishes the same postings on

A board almost never has one door. There is the results page a person sees, the endpoint that page calls to fill itself, and a syndication feed sitting beside both so partners can consume new items without walking the search.

The results page is the obvious one and the weakest. Twenty five cards per page, around 180 KB of markup for nine usable fields, and a relative posted stamp that reads "3 days ago" with no absolute timestamp anywhere in the document. Every field lives behind a selector that a template edit will move.

The endpoint the page calls is where the work belongs. On my board it answers at a search path with a page size up to 50, and it returns 31 fields against the nine the card renders. Absolute posted_at and expires_at in ISO form, a coded region alongside the free text location, an employment type, a remote flag, a company identifier, and on about two thirds of records the full description body. Finding it took one page load with the network panel open and four minutes of stripping headers until the call still answered with three of them.

The category feed is the cheapest freshness signal on the whole board. Thirty four feeds, each holding the most recent 200 items, refreshed every quarter hour, with stable identifiers and absolute stamps. It cannot backfill anything, so it never enumerates. What it does is tell me within fifteen minutes that a category has moved, which lets me schedule the intraday passes where they earn their requests.

I run all three with different jobs. The endpoint carries the enumeration. The feeds trigger intraday passes. The markup page runs on a sample of 300 records a day as a witness, because when the endpoint starts returning a renamed field the markup usually still shows the value, and comparing the two catches the rename the same morning it lands.

The depth ceiling and what the board does when you cross it

The endpoint reports a total of 412 118 and offers a page size of 50. It will not serve me page 21.

Search backends sit on an index that refuses deep offsets, and every board I have measured has a ceiling somewhere between 400 and 2 500 records per query. Mine stops at offset 1 000. The interesting part is the behaviour past that line, because a polite refusal would be easy to handle and boards are rarely polite about it. I have seen four responses to an over-deep offset: a 400 with a readable message, a 200 with an empty array, a 200 carrying the last reachable page again, and the nastiest one, a 200 carrying page one. A walker that trusts the status code will happily collect the first 50 records forty more times and report a full run.

So the walker never trusts the status code. It fingerprints the identifier set of every page and stops the moment a fingerprint repeats.

import hashlib, httpx

SEARCH = "https://board.example.net/api/jobs/search"
HEAD = {
    "accept": "application/json",
    "x-requested-with": "XMLHttpRequest",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/142.0 Safari/537.36",
}

def page_ids(client: httpx.Client, params: dict, offset: int, size: int):
    r = client.get(SEARCH, params={**params, "offset": offset, "limit": size},
                   headers=HEAD, timeout=20.0)
    r.raise_for_status()
    doc = r.json()
    return [str(x["id"]) for x in doc.get("results", [])], doc.get("total", 0)

def fingerprint(ids) -> str:
    return hashlib.blake2b(",".join(sorted(ids)).encode(), digest_size=8).hexdigest()

def reachable_depth(client: httpx.Client, params: dict, size: int = 50,
                    probe_to: int = 5000):
    """Highest offset the board serves with records it has not served already."""
    first, total = page_ids(client, params, 0, size)
    head_fp = fingerprint(first)
    lo, hi, good = 0, probe_to, 0
    while lo <= hi:
        mid = ((lo + hi) // 2 // size) * size
        ids, _ = page_ids(client, params, mid, size)
        if ids and fingerprint(ids) != head_fp:
            good, lo = mid, mid + size
        else:
            hi = mid - size
    return good, total

Eleven probe calls give me the ceiling for any filter combination, and I re-measure it once a week because boards tune this number without telling anyone. The markup list has the same limit expressed differently: page 41 renders page 1 with a 200 and no warning in the body.

Cutting one query into windows that fit under the ceiling

With 412 000 records and 1 000 reachable per query, the enumeration has to be split into at least 413 queries whose result sets do not overlap and do not leave holes. Choosing the axis is the whole problem.

Category looks tempting and fails on both counts. Postings carry two categories often enough to inflate the row count, and the board files anything ambiguous into a bucket the filter does not expose, so a category sweep loses records silently. Region fails the same way: radius search overlaps by design, and postings with no stated location vanish from every regional query. Employment type is coarse and drops the unspecified bucket.

Time partitions properly. A posting has exactly one posted_at, the board exposes posted_from and posted_to parameters, and adjacent half-open windows tile the day with no overlap and no gap. So the planner bisects the window until every piece fits.

from datetime import timedelta

CEILING = 1_000
FLOOR = timedelta(minutes=1)

def window_count(client, base: dict, lo, hi) -> int:
    p = {**base, "posted_from": lo.isoformat(), "posted_to": hi.isoformat()}
    _, total = page_ids(client, p, 0, 1)
    return total

def plan_windows(client, base: dict, lo, hi, out: list):
    n = window_count(client, base, lo, hi)
    if n == 0:
        return
    if n <= CEILING:
        out.append({"from": lo, "to": hi, "expect": n, "second_axis": None})
        return
    if hi - lo <= FLOOR:
        out.append({"from": lo, "to": hi, "expect": n, "second_axis": "category"})
        return
    mid = lo + (hi - lo) / 2
    plan_windows(client, base, lo, mid, out)
    plan_windows(client, base, mid, hi, out)

A weekday splits into 214 windows. The widest covers six hours of a quiet Saturday night, the narrowest covers four minutes of Monday morning, and the planner spends 427 count calls to work that out, each with limit=1. That is well under one percent of the day's request budget and it removes the guesswork entirely.

The floor case is real. One staffing agency pushed 3 400 records through the board inside 90 seconds, and a one-minute window still came back over the ceiling. The planner marks that window for a second axis, sweeps it by category, accepts the small gap the hidden bucket creates, and flags the run so I can go and look at it. That has happened four times in eleven months.

The fields on a posting, and the ones that move by themselves

Freshness is what people buy this data for, so the field list carries two groups: the description of the role, and the state of its life on the board. Mixing those two groups inside one hash is the mistake that makes a change detector useless.

The bump deserves its own paragraph because it dominates the daily change signal. Boards let an employer refresh a posting back to the top of the results, and a refresh moves posted_at to the current moment, extends expires_at, and touches nothing else. On my board about 9 percent of the open set gets bumped on any given day. Put posted_at in the digest and one in eleven postings reports as modified every morning for no reason. Drop the field altogether and you lose the freshness column the whole dataset exists to provide. So it lives outside the hash, in its own column, with a bump_count that increments whenever posted_at moves while the digest holds steady.

The apply link needs the same care for a duller reason. Every fetch returns a fresh tracking token in the query string, so the raw value differs on every run. I keep the path and a short allowlist of parameters that carry real routing information, drop everything else, and store the result. Description bodies get the same treatment: collapse whitespace, drop tracking pixels and identifier attributes, keep the text and the list structure.

New postings and closed ones, told apart from a failed window

Two snapshots give me three sets. Identifiers present today and absent yesterday are new. Identifiers present in both are carried forward. Identifiers present yesterday and absent today are missing, and missing is a long way from closed.

On a run where three windows died on a stalled cursor, 4 380 postings looked missing. I confirmed each one with a direct fetch of its public posting page, and 4 102 of them answered with a perfectly live page. Closing those would have written a false end date onto the largest single category on the board and poisoned every duration statistic downstream.

CLOSED_MARKERS = (
    "no longer accepting applications",
    "this position has been filled",
    "this vacancy has closed",
)

def classify_missing(client: httpx.Client, url: str) -> str:
    r = client.get(url, headers=HEAD, timeout=20.0, follow_redirects=True)
    if r.status_code in (404, 410):
        return "gone"
    if r.status_code == 200:
        body = r.text.lower()
        if any(m in body for m in CLOSED_MARKERS):
            return "closed"
        return "live"        # the enumeration missed it, leave the state alone
    return "unknown"

The state machine on top of that has four positions and a grace counter. A posting starts open. One miss moves it to suspect with misses at one. A second consecutive miss backed by a gone or closed verdict moves it to closed with a closed_at stamp. Any appearance in a later enumeration resets misses to zero, and if the posting was already closed it gets a reopened_at and keeps its identifier, because agencies reactivate expired postings and the identifier stays the same when they do.

Two extra signals feed the same decision. The expires_at field gives me an expected end date, and a posting that passes it without appearing in the enumeration is a confident closure on the first miss. The liveness probe covers the rest: every run re-checks a rotating seventh of the open set with a cheap request, so no posting goes longer than a week without a direct confirmation that it is still there. With those two in place my closure rate settled at 5.8 percent per day and the share of closures backed by a hard verdict runs above 98 percent.

The same job, posted three times, by three different names

Duplicate postings are the reason a raw job board extract is unusable as a count of open roles. Three kinds show up and they need three different treatments.

The first kind is a record collected twice inside one run, caused by overlapping windows after a mid-run replan or by a retry that succeeded twice. A primary key on posting_id and an upsert kills it at write time and it never reaches anyone.

The second kind is a repost. The same employer lets a posting expire and puts it back up a fortnight later with a new identifier and identical content. A composite key of company_id, a normalised title and the coded region catches almost all of these, and I gate it on a description similarity check so that two genuinely different openings at the same company with the same title do not merge.

The third kind is the hard one. A client hands one vacancy to four staffing agencies, and sometimes posts it directly as well. Five records, five company identifiers, five slightly different titles, and one requirement block that all of them copied from the same brief. Keys do nothing here. Near-duplicate detection over the description body does.

import re, hashlib

PUNCT = re.compile(r"[^\w\s]+", re.UNICODE)
LEAD = 60          # words of agency boilerplate to drop from the head

def to_words(text: str) -> list[str]:
    return [w for w in PUNCT.sub(" ", text.lower()).split() if len(w) > 1]

def shingles(words: list[str], k: int = 5) -> set:
    body = words[LEAD:] if len(words) > LEAD * 3 else words
    return {" ".join(body[i:i + k]) for i in range(max(0, len(body) - k + 1))}

def signature(sh: set, perms: int = 96) -> list[int]:
    MASK = (1 << 64) - 1
    sig = [MASK] * perms
    for s in sh:
        base = int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8).digest(), "big")
        for i in range(perms):
            h = (base * (2 * i + 1) + i * 0x9E3779B1) & MASK
            if h < sig[i]:
                sig[i] = h
    return sig

def bands(sig: list[int], rows: int = 4):
    for i in range(0, len(sig), rows):
        chunk = ",".join(map(str, sig[i:i + rows])).encode()
        yield hashlib.blake2b(chunk, digest_size=8).hexdigest()

def jaccard(a: set, b: set) -> float:
    inter = len(a & b)
    return inter / (len(a) + len(b) - inter) if a or b else 0.0

Ninety six permutations banded four rows at a time gives 24 bands, which pulls in candidate pairs above roughly 0.7 similarity while keeping the candidate list small enough to check exactly. Every candidate pair then gets a real Jaccard measurement and joins a cluster at 0.82 or above.

The LEAD constant is worth explaining because it doubled my recall. Agencies replace the opening paragraph with their own introduction and leave the requirement list untouched, so shingling the whole body drags similarity below the threshold on pairs that are obviously the same role to a human reader. Dropping the first 60 words before shingling took recall from 0.61 to 0.88 on a hand-labelled sample of 400 pairs, with precision holding at 0.96.

The numbers on my board: 412 000 open postings collapse into 31 700 multi-member clusters covering 78 400 records, so 46 700 rows are copies of something already present, close to 11 percent of the open set. Nothing gets deleted. Every record keeps its row and carries a cluster_id, one member per cluster is marked as the representative, and the count of agencies attached to a single vacancy turns out to be one of the more requested columns in the whole extract. Representative selection prefers the direct employer record when the cluster contains one, then the earliest posted_at.

What one daily pass actually costs

Sizing the run before it starts is arithmetic, and doing the arithmetic in advance ends the argument about why the window slipped. Below is a full weekday pass, each stage carrying its own allowance for calls that come back wrong.

Read the table for three lessons. The liveness probe accounts for 78 percent of the requests and 9 percent of the bytes, which makes it the cheapest insurance on the plan and the first line a tight window tempts me to delete. Detail work touches only a third of the new records because the endpoint already ships the description on the other two thirds, so any field promoted from the list payload takes a request out of the plan for good. The allowance of 3 to 4 percent matches what I measure against a well mannered origin at a pace it tolerates.

The first full pass is a different animal. Nine thousand three hundred and eighty enumeration calls after sharding, 142 300 detail fetches, 6.4 GB down, finished inside one overnight window. That figure is why I stopped treating volume as a planning variable at all, since the pool carries the traffic without metering it. A run of this shape is a query grinder, close enough to the way Key Collector works through a query list, and pace is the only number left to tune.

Holding a pace the board tolerates

Aggregate pace is one figure. What the origin observes is per address, and that second figure decides whether the run finishes on time or spends its window feeding a retry queue.

I hold 30 per second on this board. The bottom row reads as the quickest option right up to the moment the retry share passes 8 percent, and from there the real completion rate sinks under the 60 per second line while the board gets a sharp reading of exactly what is knocking on it. Every tunnel in the third column runs at 15 requests per minute, which is one request every four seconds from any exit the board observes, and that has stayed under the threshold on every board I have measured.

There are roughly 12 000 live addresses in the pool I draw from, and rotation happens inside it automatically, so the collector never selects an exit and never has to remember one. Concurrency and timing are mine to own. The thread figure comes off the package: a regular tier carries 1 000 concurrent connections, corporate carries 3 000, two packages do not add together, and a second source address bound in the panel splits the number in half. That leaves me 500 on a regular tier with two bindings, of which the collector claims 120 and the liveness probe claims 90, with the balance held back for whatever turns up during a pass.

import asyncio, random, time

class Throttle:
    """One aggregate ceiling, with an adaptive floor when the board pushes back."""

    def __init__(self, ceiling: float):
        self.ceiling = ceiling
        self.current = ceiling
        self.next_at = time.monotonic()
        self.gate = asyncio.Lock()
        self.last_push = 0.0

    async def slot(self):
        async with self.gate:
            now = time.monotonic()
            gap = 1.0 / max(self.current, 0.5)
            self.next_at = max(now, self.next_at) + gap
            wait = self.next_at - now
        if wait > 0:
            await asyncio.sleep(wait + random.uniform(0.0, 0.12))

    def pushed_back(self):
        self.current = max(1.0, self.current * 0.5)
        self.last_push = time.monotonic()

    def recover(self):
        if time.monotonic() - self.last_push > 60 and self.current < self.ceiling:
            self.current = min(self.ceiling, self.current * 1.2)

That jitter on the final line is nearly free and it wipes out the metronome pattern a constant sleep leaves in the timing. Every worker client is also pinned to two connections, since a pooled transport with no ceiling on it holds sockets open well past the response and silently spends threads the semaphore has already counted as available. Keeping the whole run on private server side access is what makes the latency distribution narrow enough for these thresholds to mean the same thing from one morning to the next, and sizing the term to the project with IPv4 access sold by period matches a job that runs every day for months.

Storing the history and reading what moved

A snapshot without history answers one question. A snapshot with history answers the ones people actually ask: how long a posting stays open by category, how often a title gets edited after publication, how many bumps precede a closure.

CREATE TABLE postings (
  posting_id   TEXT PRIMARY KEY,
  company_id   TEXT,
  cluster_id   TEXT,
  digest       TEXT    NOT NULL,
  state        TEXT    NOT NULL DEFAULT 'open',   -- open | suspect | closed
  misses       INTEGER NOT NULL DEFAULT 0,
  posted_at    INTEGER,
  expires_at   INTEGER,
  bump_count   INTEGER NOT NULL DEFAULT 0,
  first_seen   INTEGER NOT NULL,
  last_seen    INTEGER NOT NULL,
  closed_at    INTEGER,
  reopened_at  INTEGER,
  payload      TEXT    NOT NULL
);

CREATE TABLE posting_versions (
  posting_id  TEXT    NOT NULL,
  run_id      INTEGER NOT NULL,
  digest      TEXT    NOT NULL,
  changed     TEXT    NOT NULL,      -- json array of field names
  payload     TEXT    NOT NULL,
  seen_at     INTEGER NOT NULL,
  PRIMARY KEY (posting_id, run_id)
);

CREATE TABLE windows (
  run_id     INTEGER NOT NULL,
  win_from   INTEGER NOT NULL,
  win_to     INTEGER NOT NULL,
  expected   INTEGER NOT NULL,
  collected  INTEGER NOT NULL DEFAULT 0,
  status     TEXT    NOT NULL DEFAULT 'pending',
  leased_at  INTEGER,
  attempts   INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (run_id, win_from, win_to)
);

CREATE INDEX postings_state ON postings(state, last_seen);
CREATE INDEX versions_run   ON posting_versions(run_id);

The windows table is the resume point. Each row carries its planned count, its collected count, a lease stamp and an attempt counter, so a worker that dies holding a window releases it after fifteen minutes and whoever is free picks it up. Skip the lease and a dead worker pins its window at running for all eternity, the totals still add up on paper, and the day ships with a gap nobody sees. Those 4 380 phantom closures came from precisely that.

DIGEST_FIELDS = ("title", "company_id", "location_code", "remote_flag",
                 "employment_type", "pay_band_text", "description_norm",
                 "apply_url_norm", "source_channel")

def digest_of(row: dict) -> str:
    parts = []
    for f in DIGEST_FIELDS:
        v = row.get(f)
        if isinstance(v, (list, tuple)):
            v = "|".join(sorted(map(str, v)))
        parts.append(f + "=" + ("" if v is None else str(v)))
    blob = "\n".join(parts).encode("utf-8")
    return hashlib.blake2b(blob, digest_size=16).hexdigest()

def moved(old: dict, new: dict) -> list[str]:
    return [f for f in DIGEST_FIELDS if old.get(f) != new.get(f)]

A version row gets written only when the digest changes, and the changed column records which fields moved. Across 412 000 open postings that produces about 5 900 version rows a day, 1.4 percent, and eleven months of history sits under 4 GB in a single file with write-ahead logging on. Bumps go to the postings row alone and never create a version, which is what keeps the history table readable.

The queries this shape supports are the payoff. Median time open by category came out at 21 days with a long tail in specialist engineering. Title edits after publication affect 3.1 percent of postings, almost always once, almost always inside the first 48 hours. Median bumps before closure is two, and postings with five or more bumps close at a much lower rate, which turned out to be a decent proxy for a role nobody is filling.

Signing the morning table off

The day's snapshot counts as done once it clears the acceptance gate, which is a dozen queries with hard numbers hanging off each one. They run before anyone else opens the file.

Three of those have caught real damage. Confirmed closures dropped to 61 percent on the morning the three windows died, which stopped the run before it wrote a single end date. The bump ratio jumped to 44 percent the week a colleague added posted_at to the digest field list, and the version table would have grown by a factor of thirty in a day. Cluster share fell to 4 percent when the board switched its description body to a different markup wrapper and my normaliser started returning the navigation text for every record.

Then I do the boring part anyway. Sort by posted_at descending, read twenty postings, sort ascending, read twenty more. Broken parsing settles at the edges of a sorted column, and forty seconds spent looking there has turned up faults none of my queries were written to catch. Once the shape holds for three consecutive mornings I stop watching it daily and let the acceptance gate do the watching. Long-running collection like this stays predictable when the fetch layer does, which is why I keep it on HTTPS endpoints that carry the login pair and take the list as IP:PORT:LOGIN:PASS from the panel at the start of each pass. Two free hours are enough to plan a day of windows end to end and read the real figures off your own target, which is the sensible way to arrive at a term of access to the IPv4 pool.

Three companion pieces cover the parts this one skipped. Catalogue work with the same sharding problem and a very different freshness profile is worked through in collecting product cards in volume. Listings with thin records and heavy pagination behave closer to working through directory listings, which handles the address normalisation this job never needs. The full chain from collection to alerting lives in wiring a price monitor from collection to alert. And before fixing the pace for a daily pass of this size, the pace and pause planner turns the request count and the package ceiling into the concurrency figure to hold.