Python requests proxy session retry: pools, backoff and timeouts that hold for a long run

My first proxied script in Python was four lines long. One requests.get, one proxies dict, one print. It worked, I felt fine about it, and then I pointed the same four lines at 40 000 product URLs and watched the run die at 11 percent with a wall of red tracebacks that all said different things.

The gap between those two states is what this guide covers. A single call through an endpoint is a five minute exercise. A run that starts on Friday evening and is still collecting on Saturday morning needs a session, a sized connection pool, a retry policy that knows which answers are worth repeating, two separate timeouts, and a worker count that respects the thread limit of the package you bought. Every number below came off my own runs while writing this, against a pool of private server addresses with rotation happening inside the pool.

I will go in the order a script actually grows: the proxies dict, the session, the adapter, the retry object, the timeouts, the environment, the exit address check, the thread pool, and finally the exception names that show up in your log and what each one means.

The proxies dict, and why the key scheme decides everything

The proxies argument looks trivial and hides the single most common mistake in proxied Python. The keys are the scheme of the target URL. The value carries the scheme of the proxy transport. Two different schemes living in one short dict.

import requests

PROXY = "http://LOGIN:PASS@203.0.113.24:8000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://api.ipify.org?format=json",
                 proxies=proxies, timeout=(5, 20))
print(r.status_code, r.json())

Write only the http key and every https:// request in the script leaves through your own connection. Nothing raises. Nothing warns. The target answers normally, your log fills with 200s, and the address you were trying to keep out of the picture is the one doing the work. I found this in a run that had already made 6 000 requests, and the only reason I found it was a routine exit address check that I now run on every worker.

The value scheme picks the transport. http:// sends a plain proxy request for HTTP targets and a CONNECT tunnel for HTTPS targets. socks5:// and socks5h:// need the PySocks extra installed, which arrives with pip install requests[socks]. The trailing h moves name resolution to the proxy side, which is the form I keep in scripts, because a lookup performed on my own machine tells a local resolver every hostname the run touches.

proxies = {
    "http":  "socks5h://LOGIN:PASS@203.0.113.24:1080",
    "https": "socks5h://LOGIN:PASS@203.0.113.24:1080",
}

Keys also accept a host qualifier: https://shop.example.com as a key routes that one host through a separate endpoint while the generic https key handles the rest. I use this on jobs where one target sits behind a partner allowlist and the other 12 targets go through the shared pool. When the same credential pair has to be typed as a URI here, as the four fields A-Parser expects in its proxy list and as a --proxy argument somewhere else, I keep the shapes aligned by taking IP:PORT:LOGIN:PASS from the panel, which converts to the URI form above without guesswork.

One session, one pool, fewer handshakes

requests.get at module level builds a session, uses it once, and throws it away. Every call pays a TCP handshake, a TLS handshake, and through a proxy an extra CONNECT round trip before the first byte of your request leaves the machine. On a target with a 90 ms round trip that is roughly 400 ms of setup per call, repeated forever.

requests.Session holds three things worth having: a cookie jar, default headers, and mounted adapters that own the urllib3 connection pools. Reuse a session and the second call to the same host skips the whole setup sequence.

I measured it on a small batch to stop arguing with myself about whether it mattered. 500 GETs to one host through a server endpoint, single thread, warm DNS:

Three sockets on the middle row deserves a word. Two of them died mid batch on a rotation event inside the pool and were replaced transparently by urllib3, which is the behaviour you want and the reason the retry object further down matters more than the session itself.

One caution that costs people whole afternoons. A Session object survives concurrent use on most paths, and the cookie jar sits outside that guarantee. Sharing one session across 30 threads that all receive Set-Cookie headers produces interleaved state that is impossible to reason about. My rule is one session per thread, built through threading.local, which appears in the thread pool code later.

Sizing HTTPAdapter against the number of threads

HTTPAdapter is where a session stops being a convenience and starts being infrastructure. Two arguments carry the weight.

pool_connections is the number of connection pools kept in the cache. One pool per host, roughly. Scraping one domain needs a small number here. Scraping 200 domains in one run needs it raised, since the cache evicts the oldest pool when it overflows and every eviction throws away live sockets.

pool_maxsize is the number of connections kept inside one pool. This is the one people leave at its default of 10 while running 32 threads, and the symptom is a log line that repeats forever:

WARNING urllib3.connectionpool: Connection pool is full,
discarding connection: shop.example.com. Connection pool size: 10

That warning means every thread past the tenth opened a socket, used it, and dropped it on the floor. The run still finishes. It finishes slowly, with handshake cost on most calls, and the target sees a connection churn pattern that looks nothing like a browser.

pool_block=True changes the failure mode: a thread with no free connection waits for one. I keep it on. Blocking gives me a queue I can see in the timings, and silent socket churn gives me nothing.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def build_session(workers: int, proxy: str, hosts: int = 1) -> requests.Session:
    s = requests.Session()
    s.trust_env = False
    s.proxies = {"http": proxy, "https": proxy}
    s.headers.update({
        "Accept-Encoding": "gzip, deflate",
        "Connection": "keep-alive",
    })

    retry = Retry(
        total=4,
        connect=3,
        read=2,
        status=3,
        status_forcelist=(408, 425, 429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}),
        backoff_factor=0.6,
        respect_retry_after_header=True,
        raise_on_status=False,
    )

    adapter = HTTPAdapter(
        pool_connections=max(8, hosts),
        pool_maxsize=workers + 4,
        pool_block=True,
        max_retries=retry,
    )
    s.mount("http://", adapter)
    s.mount("https://", adapter)
    return s

The four numbers I actually deploy, taken from jobs of different shapes:

The pattern in the third column: pool_maxsize sits slightly above the worker count so no thread ever waits on a pool that could have held one more socket. The pattern in the fifth column runs the other way. High concurrency plus generous retries multiplies into a tail of work that arrives long after you thought the batch was done, so wide runs get shorter retry budgets and a second pass over the failures.

Both columns need one number from outside the script: how many parallel connections your package actually permits. The packages I run allow 1000 threads, the corporate term allows up to 3000, and they do not stack, so two packages on one account still leave you inside the higher of the two. Binding two addresses splits the allowance in half across them. Working out how many workers a given target depth needs against that ceiling is arithmetic I stopped doing by hand. The ceiling itself comes with the term, and the pages describing a proxy pool sized for collection work list the parallel connection figure for each one, which is the only number the adapter sizing above depends on.

Retry, the status list and the backoff arithmetic

urllib3.util.retry.Retry handles the repeat logic below requests, at the connection pool level, so a retried call never reaches your code. The fields that matter:

total caps every kind of retry together. connect, read and status cap their own categories under that ceiling. status_forcelist names the response codes that trigger a repeat. allowed_methods restricts repeats to methods where a repeat is safe; the default frozenset covers GET, HEAD, OPTIONS, PUT, DELETE and TRACE, with POST left out deliberately, since nobody wants a duplicated form submission.

backoff_factor produces the sleep between attempts through a doubling series: the wait is the factor multiplied by two raised to the number of previous retries. At 0.6 the gaps run about 0.6 s, 1.2 s, 2.4 s, 4.8 s. At the default of 0 there is no sleep at all, which turns a rate limited target into a machine gun pointed at your own thread budget.

respect_retry_after_header=True lets a Retry-After header override the series on 429 and 503. Targets that send that header are telling you the exact number of seconds they want, and honouring it is the difference between a temporary throttle and an escalating one.

raise_on_status=False returns the final response object with its real status code after the retries are exhausted. I want that object. A 503 body often carries the reason, and a raised RetryError carries a stack trace with nothing useful inside it.

Worst case duration per URL is worth calculating once, because it explains slow runs better than any profiler:

worst case = (total + 1) * (connect_timeout + read_timeout) + sum(backoff)
           = 5 * (5 + 25) + (0.6 + 1.2 + 2.4 + 4.8)
           = 150 + 9  = 159 seconds

One URL, 159 seconds, occupying a worker the whole time. With 24 workers and a batch of 40 000 URLs, a failure rate of 2 percent adds over an hour of pure waiting to the run. That arithmetic is why the wide profiles in the table carry total=2.

Codes I put in the list and codes I keep out:

403 and 407, the two answers that repeating never fixes

These two share a habit of appearing in bulk and they come from opposite ends of the wire, so they earn their own section.

403 comes from the target. The request reached the site, the site read it, the site declined. Repeating the same request produces the same refusal and each attempt adds another sample of the same behaviour to whatever is scoring you. Two of my worst runs died this way: a retry list with 403 in it turned a modest refusal rate into 9 000 rejected requests in eleven minutes.

What I do with a 403 is read it. The body often names the reason in plain text. The headers are more useful still, and a run that logs Server, Cf-Ray, X-Cache and Set-Cookie on refusals gives you a pattern within an hour: refusals concentrated on one path, or on requests missing a header the site expects, or on calls arriving faster than one per second from the same exit. The fix lives in the request shape and the pacing. My handler marks the URL, pauses that worker, and returns it to a slow queue that runs later with a different exit address, which works because rotation inside the pool happens on its own between calls.

407 comes from the proxy. Proxy Authentication Required means your endpoint answered and refused you. Three causes cover nearly all of them. The credential pair is wrong or carries a pasted trailing space. The machine running the script is not one of the addresses bound in the panel, which is the usual cause after moving a job from a laptop to a server. Or the package term has ended and the pool is answering with a refusal at the door.

Repeating a 407 is worse than pointless: each attempt occupies a thread slot and produces the same answer. My handler raises immediately and stops the whole batch, because a 407 on one worker means the same 407 on all of them. Binding an address takes seconds in the panel, and packages come with two bindable addresses that you can swap freely, so the recovery is a panel edit and a restart. For jobs where the script runs from a runner whose address changes, the login and password form beats binding, and both forms come with an HTTP proxy package for session work in one list.

Two timeouts, connect and read

A single number in timeout=30 sets both phases to 30 seconds. Passing a tuple splits them, and splitting them is one of the highest value edits in a scraping script.

r = s.get(url, timeout=(5, 25))   # (connect, read)

connect covers reaching the proxy and completing the handshake sequence. Through a server endpoint on the same continent this finishes in well under a second; through one on the far side of the planet it can take three. I set 5 and treat anything slower as a dead endpoint worth abandoning early.

read is the maximum gap between bytes, and this is where most people misread the argument. A response that streams for four minutes never trips a 25 second read timeout as long as data keeps arriving. A server that accepts the connection and then goes silent trips it in 25 seconds flat. Setting read low protects against silence, and it never punishes a large download.

Neither value caps total wall time for a request. Nothing in requests does. On long runs I keep a deadline per URL in my own code and abandon the future when it passes, and I keep stream=True plus an explicit size ceiling on any target that might answer with a 400 MB file when I expected a page.

trust_env and the settings that arrive from outside the script

session.trust_env defaults to True, which means requests reads proxy configuration from the environment before your run starts. HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY in both cases, .netrc for credentials, REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE for certificate verification. On Windows the same call also reads the system proxy configuration out of the registry, so a checkbox someone ticked in a control panel becomes part of your script's behaviour.

Precedence runs in this order: the proxies argument on the call, then session.proxies, then the environment. Environment values fill keys that your session left empty, which is precisely how a script with only the http key set ends up sending HTTPS through whatever the machine was configured with.

s = requests.Session()
s.trust_env = False              # no env, no .netrc, no registry
s.proxies = {"http": PROXY, "https": PROXY}

I set trust_env = False in every script that has a proxy of its own, and I set the environment variables deliberately in the shell when I want a tool with no proxy argument to follow the same route:

export HTTPS_PROXY="http://LOGIN:PASS@203.0.113.24:8000"
export HTTP_PROXY="$HTTPS_PROXY"
export NO_PROXY="localhost,127.0.0.1,169.254.169.254"

That last entry keeps metadata calls on a cloud host local. A run that sends instance metadata requests through a remote endpoint gets a hang, then a timeout, then an error message that points nowhere near the actual cause.

One more environment trap worth naming: NO_PROXY matching is suffix based on hostnames. An entry of example.com covers api.example.com, and an entry of 10.0.0.0/8 is ignored by requests, since CIDR ranges are handled inconsistently across versions.

Checking the exit address on every worker

Every run I start does the same thing before touching the target: it asks what address the world sees. The check takes one request and it has caught misconfiguration for me more often than any other habit.

def exit_ip(s: requests.Session) -> str:
    r = s.get("https://api.ipify.org", timeout=(4, 8))
    return r.text.strip()

direct = requests.get("https://api.ipify.org",
                      timeout=(4, 8)).text.strip()
through = exit_ip(build_session(8, PROXY))

assert through != direct, "traffic is leaving through the host address"
print("exit:", through)

The assertion is the point. A printed address that you glance at proves nothing at three in the morning; a failing assertion stops the run.

During the run I sample the same call once every few hundred requests from a random worker and write the answer to a counter. Two readings come out of that counter. The spread tells me how many distinct exits the run touched, which on a pool of roughly 12 000 active addresses across more than 200 countries climbs quickly. The repeat rate tells me whether a single address is carrying more of the batch than it should, which is a signal to widen the worker count so the pool has more reason to hand out fresh sockets. This is the working detail behind private server proxies built for scraping runs: the exit changes between calls on its own, so the useful measurement is the distribution over a run, taken from inside the run.

Verify the protocol too. A request to an HTTP endpoint through a SOCKS transport with the wrong scheme in the value will sometimes succeed while behaving oddly under load, and comparing headers seen by a reflection endpoint against what you sent takes ten seconds.

Threads with ThreadPoolExecutor under the package limit

Threads work well here because the workload is network bound. The GIL releases during socket waits, so 24 threads really do 24 requests at once.

import threading
from concurrent.futures import ThreadPoolExecutor, as_completed

WORKERS = 24
_local = threading.local()

def session() -> requests.Session:
    if not hasattr(_local, "s"):
        _local.s = build_session(WORKERS, PROXY)
    return _local.s

def fetch(url: str):
    try:
        r = session().get(url, timeout=(5, 25))
        return {"url": url, "code": r.status_code,
                "bytes": len(r.content), "err": None}
    except requests.exceptions.RequestException as e:
        return {"url": url, "code": None,
                "bytes": 0, "err": type(e).__name__}

results, failures = [], []
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    futures = {pool.submit(fetch, u): u for u in urls}
    for f in as_completed(futures):
        row = f.result()
        (failures if row["err"] or row["code"] != 200 else results).append(row)

print(len(results), "ok /", len(failures), "to redo")

Three details in that block carry the run. The thread local session gives every worker its own cookie jar and its own share of the pool. The single except on RequestException catches the whole family from the library, since every exception below inherits from it. And the failures land in their own list for a second pass, which recovers a good share of them without repeating the successful work.

Worker count comes from two ceilings. The first is the target: how fast you can go before refusals appear, found by walking up from 8 workers and watching the 429 rate. The second is your package: 1000 parallel connections on the terms I use, up to 3000 on the corporate term, split in half when two addresses are bound. Setting max_workers above either ceiling produces queued work that looks like slowness and reads like a network fault. Traffic volume never enters the arithmetic, since the terms carry unlimited traffic on every access period and the meter you watch is concurrency alone.

For long overnight jobs I add a threading.Semaphore around the request call to hold a ceiling on requests per second across all workers, which is gentler on the target than the same throughput delivered in bursts. Automatic rotation in the pool handles the spread of exits, so the pacing work stays where it belongs, in my own code. A schedule that runs every night sits better on a term of access measured in months, where the endpoint list and the binding survive between runs.

Library exceptions, line by line

This is the table I wish someone had handed me on my first proxied run. Every name below inherits from requests.exceptions.RequestException, and several wrap a urllib3 exception whose text appears in the message.

Two rows need a sentence more. ProxyError and ConnectionError look identical in a log and mean opposite things: the first says your endpoint answered and objected, the second says nothing answered at all. The first is a credentials or binding problem on a live machine. The second is a dead line or a wrong port, and no amount of credential editing touches it.

The other row is ProtocolError with the text Connection aborted, RemoteDisconnected. On a rotating pool this appears at a low steady rate on any long run, because a socket that was fine a moment ago belongs to an exit that has moved on. One retry clears it. Counting these per hour gives me a health number for the run: a flat rate is normal operation, and a rate that climbs through the night means the worker count outgrew what the target tolerates.

Keep a small log of the exception names by hour with the exit address attached, and after two runs the pattern speaks for itself. Mine lives in a text file with four columns, it costs nothing to write, and it has answered more questions than any monitoring stack I have tried.

If the same endpoints have to work outside Python, the neighbouring guides continue this one: the command line shape of every option above, including proxy authentication and header dumps, sits in the curl flags reference for proxied requests, the framework equivalent of sessions and retries lives in the Scrapy downloader middleware guide, and the same pool sizing arithmetic translated into JavaScript agents is covered in the Node HTTP client comparison for axios, got and undici. Before setting max_workers on a new job, run the numbers through the proxy pool and thread calculator, which turns target depth, page weight and the thread ceiling of your package into a worker count you can paste straight into the code above.