Scrapy proxy middleware: per request address selection, failure handling and the counters that prove it

Scrapy ships with proxy support already wired in. Most people meet it once, set an environment variable, watch a crawl work, and never look at the mechanism again. My own runs went that way for a long stretch, and then a job that had been finishing in 40 minutes started taking three hours with a third of the pages missing.

The cause was ordinary. Every request in that crawl was going through one address, because the built in machinery attaches an address once and keeps it, and my code had no opinion about which address that should be. Nothing was broken. The design was doing exactly what it says on the tin, and I had read the wrong promise into it.

What follows walks from that built in mechanism to a downloader middleware I wrote to replace part of it. The path runs through the meta key Scrapy reads, the middleware chain and the numbers that order it, exception handling and the second pass, the retry machinery already in the stack, pacing per address, the concurrency arithmetic against a package thread ceiling, the stats keys worth watching, and the per address journal I keep between runs. Code here comes out of jobs running on my own machines.

How Scrapy attaches an address: request.meta and HttpProxyMiddleware

The whole built in mechanism reads one key. Put a string into request.meta["proxy"] and the downloader dials that endpoint for the request.

HttpProxyMiddleware sits at order 750 in the default chain and does three small jobs. It reads http_proxy, https_proxy and no_proxy out of the environment on spider open. It fills meta["proxy"] from those variables whenever that key is absent. And it takes any user:pass@ portion out of the value, encodes it, and writes a Proxy-Authorization header, leaving a bare scheme://host:port behind in meta.

import scrapy


class PriceSpider(scrapy.Spider):
    name = "prices"

    def start_requests(self):
        for url in self.targets:
            yield scrapy.Request(
                url,
                meta={"proxy": "http://45.153.14.62:8000"},
                callback=self.parse,
            )

Two behaviours here matter more than the rest.

The first: meta survives a retry. When a request comes round again through the retry machinery, it carries the same meta dict, so the same address gets used again. If that address is the reason the request failed, every retry attempt burns against the same failing endpoint. That was my three hour crawl.

The second is a trap I stepped into while writing my first attempt. Changing meta["proxy"] later in the run leaves the old Proxy-Authorization header in place, and the built in middleware refuses to overwrite a header that already exists. So the new endpoint receives the previous endpoint's credential pair and answers 407. Any code that swaps addresses has to pop that header first.

request.headers.pop("Proxy-Authorization", None)
request.meta["proxy"] = "http://45.153.14.62:8000"

Three lines of reading beat two evenings of guessing. My list comes out of the dashboard as IP:PORT and IP:PORT:LOGIN:PASS, by link or as a file, so both shapes above are things I actually feed the parser.

A downloader middleware that picks an address per request

Once the goal becomes one decision per request, the work belongs in a downloader middleware of my own. The contract is three methods: process_request runs on the way out, process_response on the way back, process_exception when the transport itself failed.

Mine loads a list file on start, keeps a small amount of state per endpoint, and hands out the least used live address on every outbound request.

#: proxypool/middlewares.py
import logging
import random
import time
from base64 import b64encode
from collections import defaultdict
from pathlib import Path

from scrapy import signals
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)


class PoolProxyMiddleware:
    def __init__(self, crawler):
        s = crawler.settings
        path = s.get("PROXY_LIST_FILE")
        if not path:
            raise NotConfigured("PROXY_LIST_FILE is empty")
        self.crawler = crawler
        self.stats = crawler.stats
        self.scheme = s.get("PROXY_SCHEME", "http")
        self.park_seconds = s.getint("PROXY_PARK_SECONDS", 180)
        self.fail_limit = s.getint("PROXY_FAIL_LIMIT", 4)
        self.swap_times = s.getint("PROXY_SWAP_TIMES", 3)
        self.slot_per_address = s.getbool("PROXY_SLOT_PER_ADDRESS", True)
        self.entries = self._load(Path(path))
        self.parked = {}                  # endpoint -> time it comes back
        self.fails = defaultdict(int)
        self.served = defaultdict(int)

    @classmethod
    def from_crawler(cls, crawler):
        mw = cls(crawler)
        crawler.signals.connect(mw.spider_closed, signal=signals.spider_closed)
        return mw

    def _load(self, path):
        rows = []
        for line in path.read_text(encoding="utf-8").splitlines():
            line = line.strip()
            if not line or line.startswith("#"):
                continue
            parts = line.split(":")
            if len(parts) == 2:
                rows.append((f"{parts[0]}:{parts[1]}", None))
            elif len(parts) == 4:
                host, port, user, password = parts
                rows.append((f"{host}:{port}", (user, password)))
            else:
                logger.warning("skipped unparsed row: %r", line)
        if not rows:
            raise NotConfigured("proxy list parsed to zero rows")
        logger.info("proxy pool loaded: %d entries", len(rows))
        return rows

    def _live(self):
        now = time.time()
        live = [e for e in self.entries if self.parked.get(e[0], 0) < now]
        if live:
            return live
        # everything is parked: take the one closest to coming back
        return [min(self.entries, key=lambda e: self.parked.get(e[0], 0))]

    def _pick(self):
        live = self._live()
        sample = random.sample(live, min(3, len(live)))
        return min(sample, key=lambda e: self.served[e[0]])

    def _attach(self, request, endpoint, creds):
        request.headers.pop("Proxy-Authorization", None)
        request.meta["proxy"] = f"{self.scheme}://{endpoint}"
        request.meta["proxy_endpoint"] = endpoint
        request.meta.pop("proxy_reset", None)
        if creds:
            token = b64encode(f"{creds[0]}:{creds[1]}".encode()).decode()
            request.headers["Proxy-Authorization"] = f"Basic {token}"
        if self.slot_per_address:
            request.meta["download_slot"] = endpoint
        self.served[endpoint] += 1
        self.stats.inc_value("proxy/assigned")

    def process_request(self, request, spider):
        if request.meta.get("dont_proxy"):
            return None
        if "proxy" in request.meta and not request.meta.get("proxy_reset"):
            return None
        endpoint, creds = self._pick()
        self._attach(request, endpoint, creds)
        return None

Three choices in that code came out of runs that went sideways.

Random sampling of three followed by a least used pick, in place of a plain round robin, keeps the distribution even while stopping the whole crawl from marching down the list in lockstep. Pure random gave me endpoints with 200 requests sitting next to endpoints with 40 in the same run.

The dont_proxy guard exists because some requests belong on the local connection. Health probes against my own endpoint, calls to an internal queue, and the occasional page that refuses anything but a direct hit all set that key and pass through untouched.

The proxy_reset flag is how a retry asks for a new address. Without it the "proxy" in request.meta test short circuits and a retried request keeps the endpoint that already failed it. That single boolean is the difference between rotation and the illusion of rotation. Rotation also happens on the provider side, since the pool cycles addresses on its own, and pointing jobs at a pool that recycles addresses without my code asking means my list stays fresh between runs with no work from me.

Where the middleware sits, and why that number decides the behaviour

Scrapy orders downloader middlewares by integer. Outbound process_request runs in ascending order. Inbound process_response and process_exception run in descending order. Picking the wrong number produces a middleware that looks correct and fires at the wrong moment.

I settled on 743 for two reasons. Outbound, my code runs before 750, so HttpProxyMiddleware finds meta already filled and leaves it alone; the built in credential handling still works for entries I choose to pass in URI form. Inbound, 743 sits below 550, and since the return chain runs downward from 850, my process_exception fires before RetryMiddleware ever sees the failure. That ordering is what lets me park a dead endpoint and reschedule with a fresh one before the retry counter moves.

DOWNLOADER_MIDDLEWARES = {
    "proxypool.middlewares.PoolProxyMiddleware": 743,
}
PROXY_LIST_FILE = "lists/pool.txt"
PROXY_SCHEME = "http"
PROXY_PARK_SECONDS = 180
PROXY_FAIL_LIMIT = 4
PROXY_SWAP_TIMES = 3

Setting the number to 800 breaks it quietly. The built in middleware runs first, the environment variable wins, and every request in the crawl goes out through one address while my counters happily report a rotation that never happened.

Handling transport failures in process_exception and sending the request round again

A refused socket, a tunnel that never opened, a connection cut halfway through the body: none of these produce a response object. They arrive as exceptions, and process_exception is the only hook that sees them.

from scrapy.core.downloader.handlers.http11 import TunnelError
from twisted.internet.error import (
    ConnectError,
    ConnectionDone,
    ConnectionLost,
    ConnectionRefusedError,
    TCPTimedOutError,
    TimeoutError,
)
from twisted.web._newclient import ResponseNeverReceived

NETWORK_ERRORS = (
    TimeoutError,
    TCPTimedOutError,
    ConnectError,
    ConnectionRefusedError,
    ConnectionDone,
    ConnectionLost,
    ResponseNeverReceived,
    TunnelError,
)


class PoolProxyMiddleware:
    # ... continues from above

    def _park(self, endpoint, reason):
        self.fails[endpoint] += 1
        self.stats.inc_value(f"proxy/fail_reason/{reason}")
        if self.fails[endpoint] >= self.fail_limit:
            self.parked[endpoint] = time.time() + self.park_seconds
            self.fails[endpoint] = 0
            self.stats.inc_value("proxy/parked")
            logger.info("parked %s for %ds after %s",
                        endpoint, self.park_seconds, reason)

    def process_exception(self, request, exception, spider):
        if not isinstance(exception, NETWORK_ERRORS):
            return None
        endpoint = request.meta.get("proxy_endpoint")
        if endpoint is None:
            return None
        self._park(endpoint, type(exception).__name__)

        swaps = request.meta.get("proxy_swaps", 0)
        if swaps >= self.swap_times:
            self.stats.inc_value("proxy/gave_up")
            return None

        fresh = request.replace(
            dont_filter=True,
            priority=request.priority - 1,
        )
        fresh.meta["proxy_swaps"] = swaps + 1
        fresh.meta["proxy_reset"] = True
        self.stats.inc_value("proxy/swapped_on_exception")
        return fresh

Returning a Request from process_exception puts it back on the scheduler and stops the chain right there. RetryMiddleware never sees the failure, so the retry counter stays where it was and my own proxy_swaps counter carries the accounting. Keeping those two counters separate is what makes a run readable afterwards: one number for target trouble, one number for transport trouble.

dont_filter=True is mandatory here. The duplicate filter has already fingerprinted this URL, and a rescheduled request without that flag disappears into the filter with no log line at all. I lost half a day to that omission, and the symptom was a crawl that finished early with a suspiciously round item count.

The priority drop by one pushes the swapped request behind fresh work. Under a queue holding 30000 URLs that keeps a handful of stubborn pages from riding at the front of the line and slowing everything behind them.

Living with RetryMiddleware and tuning RETRY_HTTP_CODES

Transport failures are mine. Status codes belong to RetryMiddleware, which already handles them, and fighting it produces double retries that look like a pool problem.

Its defaults are worth knowing by heart: RETRY_TIMES is 2, RETRY_PRIORITY_ADJUST is -1, and RETRY_HTTP_CODES covers 500, 502, 503, 504, 522, 524, 408 and 429. Anything outside that list reaches the spider callback as a normal response.

My response hook handles the reputational codes and hands the rest back untouched.

    BAD_STATUS = {403, 407, 429, 502, 522}

    def process_response(self, request, response, spider):
        endpoint = request.meta.get("proxy_endpoint")
        if endpoint is None:
            return response
        if response.status in self.BAD_STATUS:
            self._park(endpoint, f"status_{response.status}")
            self.stats.inc_value(f"proxy/bad_status/{response.status}")
            swaps = request.meta.get("proxy_swaps", 0)
            if swaps < self.swap_times:
                fresh = request.replace(dont_filter=True)
                fresh.meta["proxy_swaps"] = swaps + 1
                fresh.meta["proxy_reset"] = True
                return fresh
        else:
            self.fails[endpoint] = 0
        return response

Adding 403 to RETRY_HTTP_CODES in settings would achieve something similar with two words of configuration. I keep it in my own hook so the same event parks the endpoint and reschedules in one place, and so the retry counters keep meaning what they say.

One line in that block earns its keep every run: the else branch that zeroes the fail count. Without it, four scattered timeouts across an hour park a healthy endpoint that recovered after the first one. Consecutive failures are a signal. Occasional failures are weather.

For jobs where the whole run is one long query loop, the accounting looks different again, and the way A-Parser meters its own threads is where I first learned to count concurrent connections per endpoint before counting requests per second.

AutoThrottle and the pace on a single address

AutoThrottle watches latency and adjusts the delay between requests. It works per download slot, and the slot key is the default reason people say it does nothing for them.

By default the slot key is the target hostname. One hostname, one slot, one delay, no matter how many addresses the requests went through. Every measurement from 30 endpoints lands in one bucket and the average tells you nothing about any of them.

The fix is one line in _attach, already present above:

request.meta["download_slot"] = endpoint

Now each endpoint carries its own slot, its own concurrency counter, and its own AutoThrottle delay. A slow endpoint gets throttled on its own without dragging the fast ones down with it.

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
AUTOTHROTTLE_DEBUG = False
DOWNLOAD_TIMEOUT = 25

AUTOTHROTTLE_TARGET_CONCURRENCY is the number that does the real work. Set to 4.0 with per address slots, Scrapy aims to keep four requests in flight against each endpoint and stretches the delay whenever latency rises. On a pool of 30 addresses that means roughly 120 requests in flight across the job, arrived at by measurement.

Turn AUTOTHROTTLE_DEBUG on for the first five minutes of any new job. It prints slot, latency and current delay on every response, and the slot column is how I confirm the per address split took effect. Reading 30 slot names go by is a satisfying way to know the wiring is right. Endpoints from a pool sized for collection jobs hold latency steady enough that AutoThrottle settles within a few hundred requests, which is the behaviour the algorithm needs to converge at all.

CONCURRENT_REQUESTS against the thread ceiling of the package

Scrapy has three concurrency knobs, and the arithmetic between them and a package thread limit is where most sizing goes wrong.

CONCURRENT_REQUESTS is the global ceiling, 16 by default. CONCURRENT_REQUESTS_PER_DOMAIN is 8 and applies per hostname. CONCURRENT_REQUESTS_PER_IP defaults to 0, and setting it above zero disables the per domain limit entirely while making DOWNLOAD_DELAY apply per target address. That last one counts the address of the target site, a distinction worth writing on a sticky note before anyone sizes a pool with it.

Now the supply side. A standard package gives 1000 threads and the corporate tier goes to 3000. Packages do not stack, so buying a second one gives a second pool with its own ceiling. A package carries 2 bound addresses, and with both machines connected at once the thread ceiling splits between them, so a two machine setup sizes against 500 apiece.

CONCURRENT_REQUESTS = 96
CONCURRENT_REQUESTS_PER_DOMAIN = 96      # per address slot, see download_slot
CONCURRENT_REQUESTS_PER_IP = 0
REACTOR_THREADPOOL_MAXSIZE = 40
DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 20000
DOWNLOAD_DELAY = 0.25
RANDOMIZE_DOWNLOAD_DELAY = True

With download_slot overridden per endpoint, the per domain setting becomes a per endpoint setting, which is why it reads 96 above. Global stays at 96 as well, and the real per endpoint number comes from AutoThrottle's target concurrency. My own margin against the thread ceiling is generous: 96 in flight against 500 available leaves room for redirects, robots fetches and the odd burst without ever brushing the limit.

REACTOR_THREADPOOL_MAXSIZE deserves a mention because its default of 10 becomes the bottleneck long before the network does. Name lookups run in that pool, and 96 concurrent requests against 10 resolver threads produces timeouts that look exactly like dead endpoints. Raising it to 40 removed a class of failure I had spent a week blaming on the pool. Transfer is not metered on any package here, so a run that pulls 80 GB of HTML in a night costs the same as one that pulls 200 MB, and access with nothing counting the bytes is what makes full catalogue passes practical.

The counters that tell you how the run actually went

Scrapy dumps its stats object at the end of every crawl. Most people skim the item count. The rows underneath carry the whole diagnosis.

The comparison I run first is proxy/assigned against downloader/request_count. Those two should sit within a few percent of each other. A large gap means requests are going out with an address my code never chose, which points at either the environment variable or a middleware order mistake.

The second comparison is retry/count against proxy/swapped_on_exception. Retries climbing while swaps stay flat means the target is pushing back on pace. Swaps climbing while retries stay flat means endpoints are dropping and the pool is doing its job. Both climbing together means the run is too fast for the current list and needs its concurrency halved.

    def spider_closed(self, spider):
        got = self.stats.get_value("proxy/assigned", 0)
        reqs = self.stats.get_value("downloader/request_count", 0)
        if reqs and abs(got - reqs) / reqs > 0.05:
            logger.warning("assigned %d addresses for %d requests", got, reqs)

A journal of failures kept per address, run after run

Stats die with the process. An endpoint that failed at the tail of last night's run is indistinguishable from a healthy one at the start of tonight's, and a fresh crawl walks straight back into it.

So the middleware writes a journal on close, and reads it on open.

import csv
from pathlib import Path

JOURNAL = Path("lists/endpoint_journal.csv")


class PoolProxyMiddleware:
    # ... continues from above

    def _read_journal(self):
        history = {}
        if not JOURNAL.exists():
            return history
        with JOURNAL.open(newline="", encoding="utf-8") as fh:
            for row in csv.DictReader(fh):
                history[row["endpoint"]] = {
                    "served": int(row["served"]),
                    "failed": int(row["failed"]),
                    "runs": int(row["runs"]),
                }
        return history

    def _write_journal(self):
        history = self._read_journal()
        for endpoint, _creds in self.entries:
            row = history.setdefault(
                endpoint, {"served": 0, "failed": 0, "runs": 0}
            )
            row["served"] += self.served[endpoint]
            row["failed"] += self.fails[endpoint]
            row["runs"] += 1
        with JOURNAL.open("w", newline="", encoding="utf-8") as fh:
            writer = csv.DictWriter(
                fh, fieldnames=["endpoint", "served", "failed", "runs", "ratio"]
            )
            writer.writeheader()
            for endpoint, row in sorted(history.items()):
                served = max(row["served"], 1)
                writer.writerow({
                    "endpoint": endpoint,
                    "served": row["served"],
                    "failed": row["failed"],
                    "runs": row["runs"],
                    "ratio": round(row["failed"] / served, 4),
                })

On start I load that file and give any endpoint with a failure ratio above 0.15 across three or more runs a lower initial position, by seeding self.served with a penalty. It stays in the list, it still gets work, and it goes last.

    def _seed_from_journal(self):
        history = self._read_journal()
        for endpoint, _creds in self.entries:
            row = history.get(endpoint)
            if not row or row["runs"] < 3:
                continue
            ratio = row["failed"] / max(row["served"], 1)
            if ratio > 0.15:
                self.served[endpoint] = int(ratio * 100)
                logger.info("penalised %s, historic ratio %.2f", endpoint, ratio)

Reading the journal weekly is a five minute habit with an outsized payoff. Endpoints cluster: when eight rows sharing the first two octets all show ratios above 0.3, the subnet is the story and I pull a fresh list from the panel. The pool behind those lines runs around 12000 active entries with the list refreshing in real time, so replacing a soured block costs one download. Pulling the IPv4 list as a plain text file drops straight into the parser at the top of this article with no format work in between, and the same parser handles the pair form when I want credentials in the file.

The other pattern the journal exposes is my own fault more often than the pool's. When ratios rise across every endpoint at once on one particular target, the address was never the problem, and the answer sits in headers, pace or session handling. I go and read a captured request before touching the list. That habit came out of comparing per thread limits the way A-Parser reports them, where the same distinction between a bad endpoint and a bad request shape shows up in the run log.

Two of my sibling guides pick up the threads this one leaves hanging: session objects, adapters and backoff outside Scrapy are covered in the requests and urllib3 retry guide, the flag by flag view of the same proxy plumbing from a shell sits in the curl proxy flags walkthrough, and the transport level equivalent in a compiled language is laid out in the Go net/http and Colly proxy guide. To turn a target's tolerance into concrete Scrapy numbers before the first crawl, the request rate planner converts a requests per minute budget and a pool size into concurrency, delay and per slot targets that drop straight into a settings file.