How to tell what blocked my request: reading the answer in the response headers

A status code gives you the verdict. The header block above it names the party that issued the verdict, and those are two different pieces of information.

For a long stretch my incident notes read like this: "403 on category pages, swapped the endpoint, traffic resumed". Notes like that record what I did without recording what happened, so the same failure came back a month later and I started from zero again. The habit that fixed it costs nothing: keep the full header block of every failed response, next to the URL and the timestamp. After a few hundred of those, the pattern became obvious. A 403 arrives from at least four different places, and each place signs its work above the body.

Four parties can stop you. The application, which knows your account and your session cookie. An edge layer in front of it, which knows your address, your handshake and your header order. A limiter, which knows only how often you asked. And your own chain, meaning the endpoint you dialled through plus whatever it announces about you on the way in. All four write headers, and none of them writes the same set.

What follows is the order I read them in, family by family, with what each family proves and the command that captures it. The last two sections cover capture technique and the table I fold a run into.

Intermediary headers: Via, X-Forwarded-For and Forwarded

The first thing I want to know is how many machines touched this exchange. Three headers answer that, and only one of them belongs in a response at all.

Via is the honest one. Any cache or gateway handling the response is supposed to append itself, so a value like Via: 1.1 varnish (Varnish/6.0) or Via: 1.1 google says a hop sits between my process and the application. Two entries mean two hops. When Via names software I did not put in the path, the error page I am reading was probably written by that software and never reached the origin.

X-Forwarded-For and Forwarded travel the other way, from client to server. They matter here because endpoints echo them back on debug routes, and because what arrives at the origin decides which address gets punished. I test that with an echo route before a run starts, never during one.

curl -i -x http://45.153.14.62:8000 https://echo.example/anything

HTTP/1.1 200 OK
content-type: application/json
via: 1.1 vegur

{
  "headers": {
    "Accept": "*/*",
    "Host": "echo.example",
    "User-Agent": "curl/8.6.0",
    "X-Forwarded-For": "45.153.14.62",
    "X-Forwarded-Proto": "https"
  },
  "origin": "45.153.14.62"
}

One address in X-Forwarded-For is the shape I want. Two addresses, with my own workstation sitting first in the list, means the endpoint appended me to the chain and handed the origin a map of where the request started. Some targets read the leftmost entry, some the rightmost, and a few feed all of them into a scoring rule, so a two entry chain turns one blocked address into two. Checking this once per package takes a minute, and it is why I stay on endpoints that add no forwarding headers for collection work.

Forwarded is the standardised version of the same idea, carrying for=, by=, proto= and host= in one line, and it shows up on Java and .NET fronts. Where both arrive at an origin, the parsers I have read prefer Forwarded and ignore the legacy line, so a sanitised X-Forwarded-For proves nothing on its own.

The trap in this family is silence. A response with none of these headers proves nothing, since a well configured edge strips them on the way out. Absence is a hint, presence is evidence.

Rate limit headers and the pace they actually permit

The limiter family is the most useful set of headers on the internet, and most scrapers throw it away. It states the exact speed the target will tolerate, in numbers, on every single response.

Two spellings are in circulation. The old one carries the X- prefix, the newer draft standard drops it and adds a policy line:

HTTP/2 200
x-ratelimit-limit: 600
x-ratelimit-remaining: 143
x-ratelimit-reset: 1786012840

HTTP/2 200
ratelimit-limit: 600
ratelimit-remaining: 143
ratelimit-reset: 47
ratelimit-policy: 600;w=3600

Read the reset field carefully, because it breaks parsers constantly. The x-ratelimit-reset value above is a unix timestamp. The ratelimit-reset value is 47 seconds from now. A parser assuming seconds sleeps for fifty six thousand years on the first dump, and one assuming a timestamp sleeps for nothing and hammers straight into a 429.

The arithmetic is short. Limit divided by window gives requests per second. One over that gives the gap for a single worker. Multiply by the workers sharing the counter and you get each worker's sleep.

def pace_from_headers(h, workers=1, safety=0.85):
    limit = int(h.get("ratelimit-limit") or h.get("x-ratelimit-limit"))
    policy = h.get("ratelimit-policy", "")
    window = int(policy.split("w=")[1]) if "w=" in policy else 3600
    per_second = (limit / window) * safety
    return workers / per_second

    # 600 in an hour, 8 workers, 15 percent held back
print(pace_from_headers({"ratelimit-limit": "600",
                         "ratelimit-policy": "600;w=3600"}, workers=8))
    # 56.47 seconds between requests per worker

The safety factor is the part I argue with people about. Running at the stated limit works until a retry or a redirect adds one unplanned request, and then the counter trips. I hold back 15 percent.

Retry-After is the limiter speaking plainly, and it arrives on 429 and on 503. It carries either a count of seconds or an HTTP date, and a parser that assumes one form mishandles the other in silence. When it says 60, I sleep 60 and change nothing else. Ignoring it is how a soft limit turns into a hard ban on the address.

One detail decides whether the arithmetic holds: what the counter is keyed on. A limit tied to an API key follows the key across every address you own, so spreading load over more endpoints buys nothing. A limit tied to the source address divides across a pool, and 600 per hour with 20 addresses becomes 12000 per hour for the job. I settle it by firing two requests from two endpoints and comparing the remaining values: matching numbers mean one shared counter, independent numbers mean the limit is per address. That second answer is what makes a wide pool of IPv4 addresses worth its keep, since a pool of roughly 12000 active entries with rotation happening automatically inside it turns a per address limit into a per job budget.

Cache and CDN headers: telling a stored copy from a fresh answer

Here is the failure that taught me to read this family first. I hit a 403, rotated to a fresh endpoint, hit the same 403, rotated again, same 403. Twenty minutes went into believing my whole package had been flagged. The header block said age: 1140: the edge had been handing me a nineteen minute old stored copy of somebody else's rejection, and my new addresses never reached the origin.

Age is the most valuable header in this family. Any value above zero means the body came out of a store, so the block you are reading may have been issued to someone else, at an address you do not control, before your run started.

curl -sI -x http://45.153.14.62:8000 https://target.example/c/tools | \
  grep -Ei '^(age|cache-control|x-cache|cf-ray|cf-cache-status|x-served-by|server-timing):'

age: 1140
cache-control: public, max-age=3600
cf-cache-status: HIT
cf-ray: 8f2a41d6ce7b1c04-FRA
server-timing: cdn-cache; desc=HIT, edge; dur=1

The vendor lines name the edge you are talking to, and each vendor names its cache states differently. Cloudflare writes cf-cache-status with values like HIT, MISS, EXPIRED and DYNAMIC, paired with cf-ray, whose suffix is the airport code of the serving site. Fastly writes x-cache: HIT, MISS, one token per tier. CloudFront writes x-amz-cf-pop.

To prove whether a block is live or stored, I break the cache key and repeat:

curl -sI -x http://45.153.14.62:8000 \
  -H 'Cache-Control: no-cache' \
  "https://target.example/c/tools?cb=$(date +%s)" | head -20

If age comes back zero and the status flips to 200, nothing ever blocked me. If age is zero and the status stays 403, the origin decided about this request, and the rest of the block is worth reading.

Vary deserves the last word here. A target that varies on Accept-Encoding and User-Agent keeps a separate stored copy per combination, which is why one worker sees a 200 and its twin sees a 403 in the same second. Aligning the agent string across a fleet removes that confusion for one line of setup.

Protection headers and the shape of a challenge page

Once age reads zero and the verdict is live, the next question is whether a protection product wrote it. These products are chatty. Almost all stamp an identifier on every response they touch, including the ones they pass, so one successful request profiles a target's defences before the run starts.

Cloudflare is the loudest. Server: cloudflare plus cf-ray appears on everything, and the line naming an active decision is cf-mitigated: challenge. A 403 carrying that line is an interactive check waiting for a browser. A 403 without it, on the same host, came from the application behind the edge.

HTTP/2 403
server: cloudflare
cf-ray: 8f2a5b19aa4d0c31-AMS
cf-mitigated: challenge
content-type: text/html; charset=UTF-8
content-length: 8114
set-cookie: __cf_bm=Yq3.dK1nT8...; path=/; max-age=1800; HttpOnly; Secure; SameSite=None

Everything in that block is diagnostic. Status 403, content type HTML on a route that normally answers JSON, a body of eight kilobytes where the real page runs to ninety, a management cookie with a thirty minute lifetime. Four signals, one conclusion, no need to read the body.

Other products sign differently. Imperva adds X-Iinfo and sets visid_incap_ and incap_ses_ cookies. DataDome adds x-datadome and a datadome cookie, and its blocks arrive as 403 with a compact JSON body. PerimeterX places _px3 and _pxhd. Sucuri stamps x-sucuri-id on everything and x-sucuri-block on what it stopped. AWS WAF is quieter: a bare 403 with x-amzn-requestid and no application headers.

The pattern worth carrying away: a protection layer answers with a small HTML body and its own cookie, while an application answers with its own content type and error shape. With both examples saved from one target, the classifier writes itself in four lines. Keeping the origin address out of the chain removes an input from these scorers before they run, which is why anonymous endpoints stay in my default profile for protected hosts.

Cookie headers and the session marker you were handed

Set-Cookie tells me whether the target thinks it has met me before. It answers a question no status code touches: does this server consider my session valid. Three behaviours cover almost everything I see.

No Set-Cookie at all means my jar is working and the server recognised the session I sent. This is the quiet, healthy case, and it should be the majority of responses in any run longer than ten requests.

A brand new session identifier on every request means my jar is being dropped. Sometimes the cause is my own code creating a fresh client per request, a two line fix. Sometimes the server refuses to bind a session to me, and a session that never sticks is itself the block, arriving with a perfectly ordinary 200 status.

max-age=0 or an expiry in the past is a deletion. The server is throwing away the session I held, and the next request gets treated as a first contact. A deletion followed by a challenge cookie is the clearest sequence in this article: something pushed me over a scoring threshold and my identity was reset on the spot.

curl -s -D - -o /dev/null -x http://45.153.14.62:8000 \
  -b jar.txt -c jar.txt https://target.example/c/tools \
  | grep -i '^set-cookie:'

set-cookie: PHPSESSID=7f1c0ab93de241; path=/; HttpOnly
set-cookie: cf_clearance=deleted; expires=Thu, 01-Jan-70 00:00:01 GMT; max-age=0

Attributes carry as much information as values. SameSite=None; Secure marks a cookie meant to survive cross site navigation, standard for management layers. A Domain=.target.example scope means the cookie follows me to every subdomain, so a session earned on the landing page carries into the API host.

The operational rule that falls out of all this is one sentence: a session belongs to one endpoint. A clearance cookie is issued against the address that earned it, and moving it to a different exit invalidates it immediately, usually with a fresh challenge on the next request. Long jobs need endpoints that hold still for the length of a session, and pinning one antidetect profile to one exit is how I hold that on the account facing part of a run.

Content-Length against the body that actually arrived

A 200 with an empty body is a block that most pipelines record as a success. This section is about catching it.

Three separate things can be true at once: what the header declares, what the socket delivered, and what your parser ended up with. Comparing the first two takes one curl flag.

curl -s -o body.html -w \
  'code=%{http_code} bytes=%{size_download} hdr=%{size_header} time=%{time_total}\n' \
  -x http://45.153.14.62:8000 https://target.example/c/tools

code=200 bytes=0 hdr=412 time=0.412

Zero bytes on a 200 is the signature of a soft block. The edge accepted the connection, answered politely and sent nothing, which keeps their logs tidy and your monitoring green. I treat any body under a threshold as a failure, and the threshold comes from the target: pull ten good pages, take the smallest, halve it.

Content-Length: 0 on a 200 is the same event, declared honestly. Content-Length larger than the bytes received is a truncation, reported by curl as error 18, with a message about a transfer closed while read data was still outstanding. That one points at the network path more than at any policy.

Transfer-Encoding: chunked removes the length header entirely, so the comparison runs against your own byte counter. Most modern origins answer this way, and a chunked response that ends after two kilobytes when it normally runs to two hundred is a truncation with no header to catch it.

The last member of this family is Content-Encoding. A body that arrives as br or gzip and gets written to disk without decoding produces a file of plausible size and no readable text, which surfaces as a parse failure downstream. My log records the declared length and the decoded length together, because the ratio is a fast check: HTML compresses to roughly a fifth, and a ratio near one means the negotiation went wrong.

Server and Alt-Svc as a hint about the stack you are talking to

Server is a small header with an outsized payoff, because it names who wrote the page you are reading. Once I know that, I know whether the refusal came from a web server, an edge product or application code, and those three call for three different moves.

Common values sort themselves quickly. nginx and openresty mean a front end that can refuse on its own, with a title tag holding the status text and a body under a kilobyte. cloudflare replaces the origin value entirely, hiding the stack behind it. awselb/2.0 is a load balancer whose 502 and 503 pages mean the target's own backend fell over. ECS with a bracketed node code is CloudFront.

The useful comparison is a success against a failure on the same host. If a 200 reports Server: nginx and the 403 on the neighbouring URL reports Server: cloudflare, two different machines are answering, and only the second has an opinion about me.

Alt-Svc sits next to it and describes protocol options:

server: cloudflare
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
x-powered-by: Express

That line advertises HTTP/3 over QUIC for the next day. Browsers act on it, and a session that starts on HTTP/2 can quietly move to HTTP/3 on the second request, changing the transport signature mid job. Command line clients hold still, which makes them a stable baseline. To find out whether the protocol is part of a verdict, I pin the version and repeat the request three times.

for v in --http1.1 --http2 --http3; do
  printf '%-10s ' "$v"
  curl -s -o /dev/null -w '%{http_code} %{http_version} %{time_total}\n' \
    "$v" -x http://45.153.14.62:8000 https://target.example/c/tools
done

A target that answers 200 on HTTP/1.1 and 403 on HTTP/2 has a rule keyed on the handshake, and that discovery has saved me more runs than any agent string edit. x-powered-by fills the last blank when it survives, naming Express, PHP or ASP.NET, which tells me the error format to expect once I reach the application.

The order of your own outgoing headers, and why it shows

Everything above reads what comes back. This section reads what you send, because half the verdicts in this article are triggered by the request itself, and the most common trigger is order.

Header order is a signature. Chrome sends its HTTP/2 pseudo headers as :method, :authority, :scheme, :path. Most command line clients send :method, :path, :authority, :scheme. The server sees that difference before a byte of your carefully written agent string is parsed.

The regular headers carry the same problem. Here is what a plain client puts on the wire:

curl -v -x http://45.153.14.62:8000 https://target.example/c/tools 2>&1 | grep '^> '

> GET /c/tools HTTP/2
> Host: target.example
> User-Agent: curl/8.6.0
> Accept: */*

Four lines. A real Chrome session on the same page sends fifteen, in a fixed order, starting with sec-ch-ua and running through sec-fetch-site, sec-fetch-mode, sec-fetch-dest and an accept-encoding listing gzip, deflate, br, zstd. Putting a Chrome agent string on that four line request produces something no browser has ever sent.

Python's default client has its own tell. It sends Accept-Encoding, Accept, Connection, then User-Agent, and it lowercases nothing, so the casing differs from a browser too. Fixing the set and the order takes a few lines:

import requests
from collections import OrderedDict

s = requests.Session()
s.headers = OrderedDict([
    ("sec-ch-ua", '"Chromium";v="126", "Not.A/Brand";v="8"'),
    ("sec-ch-ua-mobile", "?0"),
    ("sec-ch-ua-platform", '"Windows"'),
    ("upgrade-insecure-requests", "1"),
    ("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/126.0.0.0 Safari/537.36"),
    ("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
    ("sec-fetch-site", "none"),
    ("sec-fetch-mode", "navigate"),
    ("sec-fetch-user", "?1"),
    ("sec-fetch-dest", "document"),
    ("accept-encoding", "gzip, deflate, br"),
    ("accept-language", "en-US,en;q=0.9"),
])
s.proxies = {"http": "http://u17402:k93qmz@45.153.14.62:8000",
             "https": "http://u17402:k93qmz@45.153.14.62:8000"}

Three details there do the heavy lifting. The sec-fetch-* group has to match the kind of request being made, so a document navigation and a background API call carry different values, and copying one set everywhere is its own signature. The accept-encoding list has to be honest, since advertising br and failing to decode it produces the ratio problem from the previous section. And the agent version has to agree with the sec-ch-ua version.

Verification is direct: send the finished session at a header echo route and compare the printed order against a capture from a real browser on the same page. I keep both captures next to the crawler and diff them whenever a target starts refusing me for no visible reason.

Capturing headers properly: dump files, timing and a log you can search

Reading headers by eye works for one request. Everything past that needs capture that survives the run, and curl carries the flags for it.

-i prints the header block ahead of the body, fine at a terminal and terrible in a pipeline because the two get mixed. -D writes the block to its own file and leaves the body alone, and that separation makes later parsing possible.

curl -sS \
  -x http://u17402:k93qmz@45.153.14.62:8000 \
  -D "dumps/$(date +%s)-tools.head" \
  -o "dumps/$(date +%s)-tools.body" \
  -w "@curl-fmt.txt" \
  -b jar.txt -c jar.txt \
  https://target.example/c/tools >> run.log

The format file holds the measurements, one per line, so every script in the project reports the same columns:

url=%{url_effective}
code=%{http_code}
exit=%{remote_ip}
dns=%{time_namelookup}
connect=%{time_connect}
tls=%{time_appconnect}
sent=%{time_pretransfer}
first_byte=%{time_starttransfer}
total=%{time_total}
bytes=%{size_download}
redirects=%{num_redirects}

The timing phases turn a vague complaint about slowness into a location. A large time_namelookup means the resolver is the problem and the endpoint is fine. A gap between time_connect and time_appconnect is TLS negotiation, which on a proxied request includes the CONNECT tunnel coming up. A gap between time_pretransfer and time_starttransfer is the origin thinking, and no endpoint change improves it. remote_ip is the field I check most, since it prints the address curl connected to and catches a proxy setting that failed to apply in silence.

For https targets the exchange rides inside a tunnel, so the endpoint sees only the host name in the CONNECT line and the response headers arrive encrypted end to end. That is why I keep HTTPS proxy lines from the panel for anything where the header block itself is sensitive, handed out in the two shapes my scripts already parse, IP:PORT and IP:PORT:LOGIN:PASS.

One habit pays for itself within a week. Never overwrite a dump. Name each file with a timestamp and the route, keep them all, and a month later "did this target always send cf-mitigated" takes one grep.

Folding a whole run into one table

A single header block answers one question. Two hundred of them, in one table, answer the question you actually had, usually some version of "what changed".

The parser is small. Walk the dump directory, pull the fields worth comparing, print columns:

import glob, os, csv, sys

FIELDS = ["server", "cf-ray", "cf-mitigated", "age", "cf-cache-status",
          "x-cache", "retry-after", "x-ratelimit-remaining",
          "content-length", "set-cookie"]

w = csv.writer(sys.stdout, delimiter="\t")
w.writerow(["file", "code"] + FIELDS)

for path in sorted(glob.glob("dumps/*.head")):
    lines = open(path, encoding="utf-8", errors="replace").read().splitlines()
    status = next((l for l in lines if l.startswith("HTTP/")), "")
    code = status.split()[1] if len(status.split()) > 1 else "?"
    h = {}
    for line in lines[1:]:
        if ":" in line:
            k, v = line.split(":", 1)
            k = k.strip().lower()
            h[k] = (h.get(k, "") + " | " + v.strip()).strip(" |")
    body = path.replace(".head", ".body")
    size = os.path.getsize(body) if os.path.exists(body) else 0
    row = [os.path.basename(path), code] + [h.get(f, "") for f in FIELDS]
    w.writerow(row + [size])

Piped into a spreadsheet or read with column -t, that table shows three patterns no single response ever reveals.

The first is the moment of change. Sort by filename, which sorts by time, and find the row where x-ratelimit-remaining stops decreasing and retry-after appears. Everything before that row was fine, and the pace that produced it is your working pace.

The second is the split between edge and origin. Group by server and count status codes inside each group. When every 403 sits in the cloudflare group and every 200 in the nginx group, the application never refused me, and the work belongs in the handshake.

The third is the cache illusion. Filter to rows where age is above zero and count the failures living there. On the run that started this article, 34 of my 41 failures had an age above 600, so six of every seven blocks had been decided before I arrived.

One more column earns its place: the exit address, taken from remote_ip. Grouping failures by exit says whether one line went bad or the whole job did. Two failures on one address is noise. Forty spread evenly across twenty addresses is a rule about my behaviour. That distinction is why I run collection from server based endpoints out of one shared pool, where swapping a bad line means taking the next one from the list.

Reading the table takes five minutes at the end of a run, and it turns the next incident note into something worth keeping: which party refused me, on which addresses, at which pace, and whether the answer came from the origin at all. Neighbouring guides here pick the thread up, since the same header block reads differently once a browser stack generates the requests: attaching endpoints inside a browser context sits in the Playwright proxy setup guide, getting credentials accepted in a headless run sits in the Puppeteer authentication walkthrough, and pushing desktop software through the same lines sits in the Proxifier guide for Windows. When a status code needs pinning to a party before a dump exists, the response code reference table lists which codes come from an edge, which from a limiter and which from the application.