Proxy checker script: what to measure before you trust a pool list

Every collection job I run starts with a text file. A few hundred lines of IP:PORT, sometimes IP:PORT:LOGIN:PASS, pulled fresh from the panel before the run. The pool behind that file holds around 12 000 active addresses and the list regenerates on request, so the file I downloaded last night and the file I download this morning are not the same file.

The question I have to answer in the next five minutes is narrow. Which of these lines will carry work tonight, and which will waste a worker slot?

For a long time I answered it with a single curl through each address to a status page. That gave me a number I believed and a number that was wrong. One pass marked 340 entries dead. A colleague ran the same file through his own script twenty minutes later and got 31. The difference was not the pool. My script was asking one question, once, of a target that had its own opinion about scripted traffic, and calling the answer a verdict on the address.

This is the walkthrough of the checker I use now: what "alive" has to mean before you can measure it, four probes that each answer a different question, how the clock gets split, why one attempt gives you no signal, and the storage that lets today's pass be compared with yesterday's. The Python is asyncio and httpx, running as I write. There is a shell version at the end for the times you have a terminal and no environment.

What "alive" means before you write a line of code

"Alive" is not a property of an address. It is a claim about one address, at one moment, over one protocol, toward one destination. Drop any of those four qualifiers and the word stops carrying information.

A line in my list can be in any of these states at the same second: the port accepts a TCP connection, the proxy answers CONNECT with 407 because my source address is missing from the bound pair in the panel, the tunnel opens while the origin returns a challenge page carrying status 200, and the same tunnel resolves example.org fine while failing on a name that only the operator's resolver knows. A single boolean cannot hold that. My first checker printed OK or FAIL and I spent weeks chasing failures that lived in my own account settings.

So the checker starts from questions, and every question gets its own probe and its own column in the output:

1. Does the endpoint accept a TCP connection on the stated port, and how fast? 2. Does the proxy actually forward traffic, and what exit address does the far side see? 3. Does the destination I care about answer through this tunnel, with a body that contains what I expect? 4. Does name resolution work inside the tunnel, or only when I hand over a literal address?

Four questions, four answers, one row per address. A row that reads port ok / echo ok / target 429 / dns ok tells me my pacing is wrong. A row that reads port ok / echo 407 / target skipped / dns skipped tells me to open the panel and check which of my two bound addresses is stale. Those are different working days, and the old boolean flattened both into FAIL.

Why a single request settles nothing

A request through a proxy crosses at least three failure domains: my own egress, the operator's hop, and the destination. Each has its own noise. A single sample cannot separate signal from noise in any of them.

Run the numbers. Say an address genuinely succeeds 90 percent of the time, which for collection work is fine. One probe returns a failure 10 percent of the time, and if I act on that probe I discard a good address once in ten passes. Across a list of 400 entries that is 40 wrongly discarded per pass. Now go the other way: an address that succeeds 30 percent of the time still passes a single probe almost a third of the time, so it stays in my working list and poisons the run with retries.

Five attempts change the picture. A 90 percent address fails all five once in a hundred thousand passes. A 30 percent address clears five out of five about twice in a thousand. The cost is five times the sockets and five times the seconds, which is why the series runs only against the probes that matter and only for entries that cleared the cheap ones.

There is a second reason a single request lies, and it took me longer to accept. The first request through a fresh tunnel is the slowest one by a wide margin: connection setup, TLS handshake, sometimes a redirect the origin only serves to new callers. If I time one request per address and rank the list by that number, I am ranking handshake luck. The series gives me a first-attempt time and a warm time, and the gap between them is itself a useful metric.

Four probes that answer four different questions

Each probe is a separate coroutine with its own timeout, and each returns a small record with the outcome, the elapsed milliseconds and a short note. Nothing raises out of a probe; a probe that cannot answer returns a record saying so.

The port probe never touches HTTP. It opens a socket and closes it.

import asyncio, time

async def probe_port(host: str, port: int, timeout: float = 4.0) -> dict:
    t0 = time.perf_counter()
    try:
        fut = asyncio.open_connection(host, port)
        reader, writer = await asyncio.wait_for(fut, timeout=timeout)
        writer.close()
        await writer.wait_closed()
        return {"probe": "port", "ok": True, "ms": _ms(t0), "note": ""}
    except asyncio.TimeoutError:
        return {"probe": "port", "ok": False, "ms": _ms(t0), "note": "timeout"}
    except OSError as e:
        return {"probe": "port", "ok": False, "ms": _ms(t0), "note": type(e).__name__}

def _ms(t0: float) -> int:
    return int((time.perf_counter() - t0) * 1000)

The echo probe answers the question the port probe cannot: does traffic come out the other side, and where. I run my own echo on a small box, because public echo endpoints rate limit exactly the kind of burst a checker produces. The whole server side is six lines of nginx:

location = /ip {
    default_type application/json;
    return 200 '{"ip":"$remote_addr","ts":"$msec","ua":"$http_user_agent"}';
}

That gives me the exit address as the far side sees it, which is the only honest way to confirm that rotation inside the pool is doing its work. Collecting the distinct exit addresses across a pass is a metric in its own right, and on my last pass over 400 entries I counted 388 distinct exits.

The target probe is the one that decides whether tonight's job runs. It goes to the actual origin I plan to collect from, with the same headers my collector sends, and it asserts on the body. Status 200 with 900 bytes of challenge HTML is a failure wearing a success code, so the probe checks for a marker string that only appears on a real page.

import httpx

async def probe_target(client: httpx.AsyncClient, url: str, marker: str) -> dict:
    t0 = time.perf_counter()
    try:
        async with client.stream("GET", url) as r:
            ttfb = _ms(t0)
            body = await r.aread()
        text = body.decode("utf-8", "ignore")
        ok = r.status_code == 200 and marker in text
        return {"probe": "target", "ok": ok, "ms": _ms(t0), "ttfb": ttfb,
                "code": r.status_code, "bytes": len(body),
                "note": "" if ok else ("marker missing" if r.status_code == 200 else "code")}
    except httpx.ProxyError as e:
        return {"probe": "target", "ok": False, "ms": _ms(t0), "code": 0, "note": f"proxy:{e}"}
    except (httpx.ConnectError, httpx.ReadTimeout, httpx.RemoteProtocolError) as e:
        return {"probe": "target", "ok": False, "ms": _ms(t0), "code": 0, "note": type(e).__name__}

The fourth probe is the one most checkers skip, and it explains a failure class that otherwise looks like magic. When a client sends a hostname to a SOCKS5 endpoint, the endpoint resolves it. When it sends a literal address, no resolution happens anywhere in the tunnel. So I run the same request twice: once by name, once by literal address with an explicit Host header, using an address I resolved myself before the pass started.

async def probe_dns(client: httpx.AsyncClient, host: str, ip: str, path: str = "/ip") -> dict:
    t0 = time.perf_counter()
    by_name = by_ip = None
    try:
        by_name = (await client.get(f"https://{host}{path}")).status_code
    except Exception as e:
        by_name = 0
    try:
        r = await client.get(f"https://{ip}{path}", headers={"Host": host})
        by_ip = r.status_code
    except Exception:
        by_ip = 0
    ok = by_name == 200
    note = "resolver inside tunnel" if (by_ip == 200 and by_name != 200) else ""
    return {"probe": "dns", "ok": ok, "ms": _ms(t0), "note": note}

If the literal form answers and the name form does not, the tunnel carries packets and the operator's resolver is the piece that stalled. That is a completely different ticket from a dead port, and until I added this probe those rows sat in my log as unexplained timeouts. The same split shows up when I put a job behind an HTTP endpoint that resolves at the hop, where the collector resolves nothing locally and every name goes over the connection.

Splitting the clock, because one number hides the problem

A single "response took 1 840 ms" tells me nothing actionable. I need to know which phase ate it, and I can get a usable split without patching the transport.

Three measurements give me four phases. The port probe already gives t_tcp, the round trip to the endpoint itself. Streaming the target request gives t_headers from the first byte written to the response headers arriving. Reading the body gives t_body. Subtraction gives the rest:

• t_tcp: my machine to the endpoint. Distance and path quality live here.
• t_setup = t_headers - t_tcp - t_origin_est: tunnel establishment plus TLS to the origin.
• t_ttfb = t_headers: everything up to the first response byte, the number that decides how many pages per minute a worker manages.
• t_body: transfer of the payload, which scales with page weight and tells me whether compression negotiation worked.

I record percentiles, never averages. One address sitting at 4 seconds drags a mean of 60 samples upward by 60 ms and nobody notices; the same address moves p95 visibly. My working thresholds after several months of passes: t_tcp p50 under 120 ms, t_ttfb p50 under 700 ms, t_ttfb p95 under 1 900 ms. Entries above those still work, and I keep them in a second tier for jobs where latency does not matter.

Two habits from experience. Time the second attempt separately from the first, because handshake cost belongs in its own column. And record the clock skew of your own machine at the start of the pass, since a checker that ran during an NTP correction produced a set of negative durations I stared at for an hour.

Success rate over a series, and how long the series has to be

The output of my checker is not a boolean. It is a ratio and a count: 4/5, 9/10, 2/5. Ordering by that ratio, with p95 latency as the tiebreak, gives the working list for the night.

Series length is a budget question. Every attempt costs a socket and a second, and my package hands me a fixed number of concurrent connections, so a longer series over the same pool sample means a longer pass. The numbers I settled on:

• Cheap probes (port) run once. A refused socket that answers on retry within the same pass is noise I do not care about.
• Echo runs twice. Once to confirm forwarding, once to see whether the exit address changed between calls.
• Target runs five times with a 400 ms pause between attempts, which is the smallest series that separates a 60 percent address from a 90 percent one with any confidence.
• The DNS pair runs once, and only for entries where echo succeeded.

Five attempts give a resolution of 20 percentage points, which sounds coarse until you consider what the ratio is for. I am sorting entries into three buckets: take now, hold in reserve, re-check later. Three buckets need much less precision than a percentage does. When I want a real figure for a specific address, I run a dedicated 60-attempt series against it and let it take a minute.

Sampling matters as much as series length. Checking all 12 000 entries on every pass is pointless work; I sample 400 per pass, weighted so that entries used in last night's job get re-checked first and the rest rotate through over the day. That keeps a pass under two minutes and still touches the whole list across a working day.

The metrics I record on every pass

Everything below goes into storage on every pass. The columns exist so that a row can be read six weeks later without any memory of the run that produced it.

The last row is the one I check first each morning. An entry that flips between passes is worse for a long job than an entry that is consistently unavailable, because the consistent one gets skipped once and the flipping one takes a worker down mid-run.

The checker itself, with a ceiling on concurrency

Here is the core of the script. It reads the list, builds one client per entry, runs the probes under a semaphore and writes rows into SQLite. I have trimmed the argument parsing and the logging setup.

import asyncio, sqlite3, time, json, statistics
import httpx

ECHO = "https://echo.mylab.dev/ip"
TARGET = "https://books.toscrape.com/catalogue/page-2.html"
MARKER = "product_pod"

def parse(line: str):
    p = line.strip().split(":")
    if len(p) == 2:
        return {"host": p[0], "port": int(p[1]), "url": f"http://{p[0]}:{p[1]}"}
    if len(p) == 4:
        return {"host": p[0], "port": int(p[1]),
                "url": f"http://{p[2]}:{p[3]}@{p[0]}:{p[1]}"}
    return None

async def check_one(entry: dict, sem: asyncio.Semaphore) -> dict:
    async with sem:
        row = {"host": entry["host"], "port": entry["port"], "attempts": []}
        port = await probe_port(entry["host"], entry["port"])
        row["port"] = port
        if not port["ok"]:
            return row

        limits = httpx.Limits(max_connections=2, max_keepalive_connections=1)
        timeout = httpx.Timeout(connect=5.0, read=8.0, write=5.0, pool=5.0)
        async with httpx.AsyncClient(proxy=entry["url"], limits=limits,
                                     timeout=timeout, follow_redirects=False,
                                     headers={"user-agent": UA}) as client:
            echo = await probe_echo(client, ECHO)
            row["echo"] = echo
            if not echo["ok"]:
                return row
            for i in range(5):
                row["attempts"].append(await probe_target(client, TARGET, MARKER))
                await asyncio.sleep(0.4)
            row["dns"] = await probe_dns(client, "echo.mylab.dev", ECHO_IP)

        ok = [a for a in row["attempts"] if a["ok"]]
        row["rate"] = len(ok) / 5
        if ok:
            row["ttfb_p50"] = statistics.median(a["ttfb"] for a in ok)
        return row

async def main(path: str, concurrency: int):
    entries = [e for e in (parse(l) for l in open(path)) if e]
    sem = asyncio.Semaphore(concurrency)
    t0 = time.perf_counter()
    rows = await asyncio.gather(*(check_one(e, sem) for e in entries))
    save(rows, elapsed=int(time.perf_counter() - t0))

The concurrency argument is the part people get wrong, and getting it wrong is how a checker takes down the job it was supposed to protect. My package gives a fixed thread count: 1 000 on the regular tiers, up to 3 000 on the corporate one. Packages do not stack, and binding a second source address splits the count in half, so with two bound addresses I have 500 to spend. The checker is not the reason I bought those threads, so it gets a slice:

checker_concurrency = floor(package_threads / bound_ips * 0.25)

With 1 000 threads and 2 bound addresses that gives 125, and a pass over 400 entries with a five-attempt series finishes in about 90 seconds. When the checker runs beside a live collector I drop the share to 0.1 and accept the longer pass. Running server side hardware through an IPv4 pool with SOCKS5 access makes this arithmetic straightforward, since the thread count is stated per package and the same ceiling applies to the checker and to the job.

One detail that bit me: max_connections=2 per client. Without it, httpx pools connections aggressively and a single misbehaving entry holds sockets open long after its probes returned, which quietly eats the budget the semaphore thinks it is protecting.

Reading a refusal correctly, and the false positives that follow

Most of the value in a checker sits in this table. Each signature below maps to a different owner: my account, the hop, or the destination. Mixing them is what produces a list of "dead" addresses that are working fine for everyone else.

Three false positives caused me more wasted work than everything else combined.

The first was my own target. I pointed the target probe at a site that serves a different page to callers it does not recognise, and my marker string lived only on the recognised version. The checker reported a pool-wide collapse. The fix was to assert on a marker that appears on both variants and record the byte length separately.

The second was IPv6 on my own machine. My resolver handed back an AAAA record, my egress had no working v6 path, and the failure surfaced as a timeout that looked exactly like a dead endpoint. Forcing the family in the client removed a whole column of phantom failures.

The third was concurrency on the destination. At 125 concurrent probes I was sending my whole checker at one origin in a burst, and the origin answered with 429 for everything after the first forty. The checker blamed the addresses. Spreading target probes across four destinations and adding jitter to the series fixed it, and the block-header reading I do when a code looks ambiguous saved me from repeating that mistake elsewhere.

A shell pass when there is no environment

Sometimes I am on a box with curl and nothing else. This gets me the same first two probes and a timing split, with xargs supplying the parallelism.

#!/usr/bin/env bash
#call it as: ./quickcheck.sh list.txt 40
LIST="$1"; PAR="${2:-20}"
TARGET="https://books.toscrape.com/catalogue/page-2.html"
FMT='%{http_code} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total} %{size_download}'

probe () {
  line="$1"
  IFS=':' read -r ip port user pass <<< "$line"
  if [ -n "$user" ]; then px="http://$user:$pass@$ip:$port"; else px="http://$ip:$port"; fi
  out=$(curl -sS --max-time 10 --proxy "$px" -o /dev/null -w "$FMT" "$TARGET" 2>/dev/null)
  [ -z "$out" ] && out="000 0 0 0 0 0"
  echo "$ip:$port $out"
}
export -f probe; export TARGET FMT

xargs -a "$LIST" -I{} -P "$PAR" bash -c 'probe "$@"' _ {} \
  | sort -k2,2 -k5,5n \
  | tee "pass-$(date +%H%M%S).txt"

The output columns line up with the Python fields: time_connect is t_tcp, time_appconnect covers the TLS handshake through the tunnel, time_starttransfer is TTFB, and size_download catches the short-body case that a status code hides. Sorting by code then by total time puts the usable entries at the top of the file. Two things this version does not give me: the exit address, and any series at all. It answers "is the file broadly sane" in twenty seconds, and I treat its output as a triage pass before the real one.

Storing passes so today can be compared with yesterday

A checker that prints to the terminal is a checker whose results die with the terminal window. Mine writes SQLite, one file per week, two tables.

CREATE TABLE IF NOT EXISTS passes (
  pass_id    INTEGER PRIMARY KEY,
  started_at INTEGER NOT NULL,
  sample     INTEGER NOT NULL,
  seconds    INTEGER NOT NULL,
  note       TEXT
);

CREATE TABLE IF NOT EXISTS results (
  pass_id  INTEGER NOT NULL REFERENCES passes(pass_id),
  addr     TEXT    NOT NULL,
  rate     REAL,
  t_tcp    INTEGER,
  ttfb_p50 INTEGER,
  ttfb_p95 INTEGER,
  code     INTEGER,
  exit_ip  TEXT,
  dns_ok   INTEGER,
  note     TEXT,
  PRIMARY KEY (pass_id, addr)
);
CREATE INDEX IF NOT EXISTS results_addr ON results(addr);

The comparison between two passes is a single query, and it is the first thing I look at:

SELECT a.addr,
       b.rate AS was, a.rate AS now,
       a.ttfb_p50 - b.ttfb_p50 AS ttfb_delta,
       a.note
FROM   results a
JOIN   results b ON b.addr = a.addr AND b.pass_id = :prev
WHERE  a.pass_id = :cur
  AND  (abs(a.rate - b.rate) >= 0.4 OR a.ttfb_p50 - b.ttfb_p50 > 500)
ORDER  BY ttfb_delta DESC;

Two passes tell me about entries. Thirty passes tell me about the pool, and that is the part worth the storage cost. From my own history: the median TTFB across a week moves by under 80 ms, so a morning where it jumps 400 ms is about my side or about the destination, and I have stopped opening tickets over it. Entries that flip more than four times in a week get parked in a separate table and never enter a long job. The distinct-exit count per pass is the number I watch for pool health, since a run over a private pool of server side addresses should return nearly as many distinct exits as probes sent, and a sudden drop in that ratio is visible long before any single job fails.

What I do with the ranked output is ordinary. The top bucket, rate 0.8 and above with p95 under 1 900 ms, goes straight into the worker queue as this run's working list. The middle bucket is held for retries when a worker exhausts its primary entry. The bottom bucket is written back with a re-check timestamp and skipped for the next four hours. The job runner reads only the working list, so a bad pass degrades throughput and never stops the run. Since the file regenerates on demand and traffic is unmetered on every tier, the checker can afford to be generous with probes, and that is the argument for packages with unmetered traffic when the pass runs hourly. For long-running collectors I keep the checker on a schedule beside the job, pulling from addresses that sit in data centre racks so that the latency distribution stays narrow enough for the thresholds above to mean something, and a short trial window is enough to record two or three passes and see the shape of the numbers before committing to a tier. The measurements are yours after that: nobody else's benchmark page knows what your origin does at your pace, and private endpoints on owned hardware plus your own numbers beat any published figure.

If you want to go deeper on the pieces this article touches, my write-up on reading the headers behind a block covers the header sets that separate a hop fault from an origin decision, which is the distinction the failure table above depends on. When a probe result makes no sense at all, inspecting proxied traffic with mitmproxy shows how to put a recording layer between the collector and the tunnel and read the actual bytes. Teams that want this pass running unattended will find the scheduling and secret handling in running proxied jobs on CI runners. And before you set the semaphore value, the pool and thread calculator works out how much of your package the checker can borrow without starving the collector next to it.