mitmproxy inspect requests: reading what your client puts on the wire

The request that leaves your socket and the request your code describes are two different objects. Libraries add headers you never set, reuse connections you thought were closed, and fire calls you never wrote. Until you watch the wire, every theory about a refusal is a guess.

I lost the better part of a week on a target that answered 403 to my collector and 200 to Chrome on the same machine, through the same endpoint, with the same agent string and the same cookie jar. Reading my own code told me nothing, because the code was correct. Putting mitmproxy between the collector and the endpoint took eleven minutes and answered the question on the first capture. My seventeen intended requests had arrived as forty one. Three of the extra ones carried a duplicated Cookie header, and one of them was a preflight my HTTP library fired on every POST without mentioning it anywhere in the documentation I had read.

This guide walks the path I now follow on any unexplained refusal: get the interceptor running, get the certificate trusted, put the working endpoint behind it so the capture shows the real chain, filter down to the flows that matter, then let a Python addon write the findings out so the next incident starts from evidence.

What turns up in your own outgoing traffic

Before the commands, the payoff. Six things show up in almost every first capture I run for someone, and none of them are visible from inside the application code.

Header count is the loudest. A requests session that looks like four headers in your editor puts eight on the wire once the adapter, the cookie jar and the compression negotiation have added theirs. Header order is the second: your library writes them in insertion order or alphabetically, and neither matches a browser.

Duplicate cookies come third, and they are the quietest failure in the list. Set a cookie manually in headers while a jar is also active and both go out, in two separate Cookie lines or one merged line with the same key twice. Some origins take the first value, some the last, some reject the request outright.

Then come the calls you never wrote: preflight OPTIONS, redirect chains your client follows silently, retry attempts from an adapter, favicon and manifest fetches from a browser context, and telemetry from whatever SDK is linked in. My record for a job that was supposed to be one request per page is nine.

The fifth is the connection story. Keep alive means twenty requests share one TLS handshake, so a target that scores handshakes sees one client and a target that scores request rate sees twenty. The sixth is protocol drift: a client that starts on HTTP/2 and falls back to HTTP/1.1 on a retry changes its whole signature mid job.

Getting mitmproxy running as an ordinary intermediary

Three binaries come out of one install and they share every flag. mitmproxy is the terminal interface, good for poking at a live flow. mitmweb opens the same thing in a browser tab. mitmdump prints and writes without a UI, which is what belongs in a script.

pip install mitmproxy
mitmproxy --version

Start the interface on a port nothing else wants. I use 8081 because 8080 is usually taken by something already running.

mitmproxy --listen-host 127.0.0.1 --listen-port 8081

The headless twin, writing every flow to a file for later reading:

mitmdump --listen-host 127.0.0.1 --listen-port 8081 -w run.flows

Nothing appears until a client dials in. Point one at it:

curl -x http://127.0.0.1:8081 -sI https://target.example/c/tools

For a Python collector the change is two lines, and both schemes point at the same interceptor because mitmproxy handles the tunnel itself:

proxies = {
    "http":  "http://127.0.0.1:8081",
    "https": "http://127.0.0.1:8081",
}
r = requests.get("https://target.example/c/tools", proxies=proxies, timeout=15)

Browsers and desktop software take the same address through their own settings. Anything reading the environment picks it up from two variables:

export HTTP_PROXY=http://127.0.0.1:8081
export HTTPS_PROXY=http://127.0.0.1:8081

The -n flag in that last row means no listening socket at all, and -r reads a saved file. Pairing them is how I re-run a new parser against last month's evidence without touching the target again.

The certificate, and the capture you get without it

Skip the certificate and your https flows arrive as a single line each: CONNECT target.example:443. That is the whole record. The client opened a tunnel, refused the interceptor's certificate, and everything after the handshake stayed encrypted between the client and the origin. You learn which hosts were contacted and how often, and nothing else.

The certificate authority is generated on first launch and lives in ~/.mitmproxy/. Trust it in the place your client actually looks, since operating system trust and library trust are separate stores.

ls ~/.mitmproxy/
#mitmproxy-ca-cert.pem  mitmproxy-ca-cert.cer  mitmproxy-ca.pem

Per client, one line each:

#curl
curl --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem -x http://127.0.0.1:8081 https://target.example/

#python requests and anything on certifi
export REQUESTS_CA_BUNDLE=$HOME/.mitmproxy/mitmproxy-ca-cert.pem
export SSL_CERT_FILE=$HOME/.mitmproxy/mitmproxy-ca-cert.pem

#node
export NODE_EXTRA_CA_CERTS=$HOME/.mitmproxy/mitmproxy-ca-cert.pem

#go, which reads the system store on linux
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy.crt
sudo update-ca-certificates

On Windows the store lives in the registry and certutil writes to it:

certutil -addstore -f "ROOT" %USERPROFILE%\.mitmproxy\mitmproxy-ca-cert.cer

There is also the browser route, which works when a machine has no shell access worth using: with the intermediary configured, open http://mitm.it and pick the platform. The page only renders when traffic is flowing through mitmproxy, which makes it a working check on the setup by itself.

Two clients will refuse anyway. Anything with a pinned certificate compares the chain against a copy compiled into the binary, and a packaged desktop client built that way will drop the connection with no useful message. Anything doing mutual TLS presents its own certificate to the origin, so the interceptor has to be handed the key pair with --set client_certs=dir. For everything else, a trusted authority turns the CONNECT line into a full request and response pair with headers, body and timings.

Upstream mode, where the real chain becomes visible

Here is the part most walkthroughs skip. Testing against a bare interceptor tells you what your client sends to 127.0.0.1. The question that matters is what your client sends through the endpoint you actually run in production, and those are not the same capture: an intermediary can add headers, rewrite the request line, or answer for itself.

Upstream mode chains them. mitmproxy accepts your client, then dials your working endpoint as its own next hop, so you see both sides of the middle.

mitmdump -p 8081 \
  --mode upstream:http://45.153.14.62:8000 \
  --upstream-auth u17402:k93qmz \
  -w chain.flows

Your collector keeps pointing at 127.0.0.1:8081 and changes nothing else. The endpoint credentials move into --upstream-auth, which sends them as Proxy-Authorization on the outer hop and keeps them out of the inner request where a target could read them.

This is how I answer the three questions that come up on every chain:

#1. does the exit address match what the panel handed me
curl -x http://127.0.0.1:8081 -s https://echo.example/ip

#2. does anything get appended to the request on the way through
curl -x http://127.0.0.1:8081 -s https://echo.example/headers

#3. does the endpoint answer for itself on failures
curl -x http://127.0.0.1:8081 -sI https://nonexistent.invalid/

The second one is where surprises live. A forwarding header added at the middle hop turns one blocked address into two, since the origin now holds the address the request started from as well as the one it arrived from. I check that once per package and then stop worrying about it, because exits that announce nothing about the client hand the origin exactly one address to score, and a single flow in the capture settles the question for good.

If the upstream itself speaks https, add --ssl-insecure while testing so a self signed certificate on the middle hop does not stop the run before you get to read anything. Take it back out afterwards.

Filters, and finding the flows that matter

A five minute capture of a browser session runs to several hundred flows. The filter language is short and it is the same in the terminal interface, in the web UI and on the command line, which means an expression you work out interactively goes straight into a script.

Two of these run on saved files, which is where they earn their keep:

#every refusal from one host, written to its own file
mitmdump -nr run.flows "~d target.example & ~c 403" -w blocked.flows

#everything my client sent that carried a body
mitmdump -nr run.flows "~m POST | ~m PUT" --flow-detail 3

#the calls that never came back
mitmdump -nr run.flows "~q & !~s"

In the terminal interface, f opens the filter prompt and the same expression applies live. Z clears the view so a fresh action starts from an empty list, which is the fastest way to isolate one button click on a page. --flow-detail 3 prints full headers, and level 4 adds bodies.

A Python addon that writes the findings for you

Reading flows by eye works for one problem. Anything recurring belongs in an addon, which is a plain Python file mitmproxy loads with -s. Hooks are named after events, and the two that carry the work are request and response.

The addon below records the outgoing header order, flags duplicated cookie keys, counts calls per host, and writes one JSON object per flow so the output can be grepped or loaded into a table later.

#findings.py
import json, time
from collections import Counter
from mitmproxy import http

OUT = open("findings.jsonl", "a", encoding="utf-8")
SEEN = Counter()

def request(flow: http.HTTPFlow) -> None:
    flow.metadata["t0"] = time.time()
    SEEN[flow.request.host] += 1

def response(flow: http.HTTPFlow) -> None:
    req, res = flow.request, flow.response
    names = [k.lower() for k, _ in req.headers.items(multi=True)]

    cookie_keys = []
    for raw in req.headers.get_all("cookie"):
        for part in raw.split(";"):
            if "=" in part:
                cookie_keys.append(part.split("=", 1)[0].strip())
    dupes = [k for k, n in Counter(cookie_keys).items() if n > 1]

    row = {
        "host": req.host,
        "method": req.method,
        "path": req.path.split("?")[0],
        "code": res.status_code,
        "http_version": res.http_version,
        "order": names,
        "header_count": len(names),
        "dup_cookies": dupes,
        "duplicate_cookie_header": names.count("cookie") > 1,
        "set_cookie": len(res.headers.get_all("set-cookie")),
        "bytes": len(res.content or b""),
        "ms": round((time.time() - flow.metadata["t0"]) * 1000),
        "hits_on_host": SEEN[req.host],
    }
    OUT.write(json.dumps(row, ensure_ascii=False) + "\n")
    OUT.flush()

def done():
    for host, n in SEEN.most_common():
        print(f"{n:5d}  {host}")

Live, in front of the endpoint:

mitmdump -p 8081 --mode upstream:http://45.153.14.62:8000 \
  --upstream-auth u17402:k93qmz -s findings.py -q

Offline, against a capture recorded earlier:

mitmdump -nr run.flows -s findings.py -q

The done hook prints the per host tally on exit, and that tally is where the calls you never wrote come to the surface. Reading the file afterwards is one line of shell:

jq -r 'select(.duplicate_cookie_header or (.dup_cookies|length>0))
       | [.code, .path, (.dup_cookies|join(","))] | @tsv' findings.jsonl

Two more hooks are worth knowing. requestheaders fires before the body arrives, which is where you skip large uploads. error fires when a flow dies without a response, and logging it separately stops timeouts from vanishing out of your statistics.

What I read first in a capture

Order matters here, because the cheap checks eliminate most causes before the expensive ones start. This is the sequence, and the reasoning behind each step.

The header order line deserves the extra sentence. Chrome sends its HTTP/2 pseudo headers as :method, :authority, :scheme, :path, and most command line clients send :method, :path, :authority, :scheme. That difference reaches the origin ahead of any header you wrote. mitmproxy shows the real sequence in the flow detail view, which makes it the only place I trust to confirm a fix landed.

For collection work I keep the whole capture running against the same package I will use in production, because a header set proved through one kind of endpoint proves nothing about another. Running the check through endpoints tuned for collection jobs means the capture and the run share a pool, with rotation happening automatically inside roughly 12000 active entries, so what I measured is what the target will see.

Comparing your client with a browser, line by line

The comparison is the whole method, and it needs both sides captured through the same interceptor within a few minutes of each other. Point Chrome at 127.0.0.1:8081, load the page by hand, then run your collector at the same URL. Two flows, one file.

mitmdump -p 8081 -w compare.flows

Then pull the two header lists out and diff them:

#diffheads.py, run as: mitmdump -nr compare.flows -s diffheads.py -q
from mitmproxy import http

CAPTURED = {}

def request(flow: http.HTTPFlow) -> None:
    ua = flow.request.headers.get("user-agent", "")
    tag = "browser" if "Chrome" in ua and "python" not in ua.lower() else "client"
    CAPTURED.setdefault(tag, []).append(
        [k.lower() for k, _ in flow.request.headers.items(multi=True)]
    )

def done():
    b = CAPTURED.get("browser", [[]])[0]
    c = CAPTURED.get("client", [[]])[0]
    width = max(len(b), len(c))
    print(f"{'browser':38} {'client':38}")
    for i in range(width):
        left = b[i] if i < len(b) else ""
        right = c[i] if i < len(c) else ""
        mark = " " if left == right else "*"
        print(f"{mark} {left:36} {right:36}")
    print("\nbrowser only:", sorted(set(b) - set(c)))
    print("client only: ", sorted(set(c) - set(b)))

The output puts the two sequences side by side with a star on every row that differs. On my collector, the run that started this article printed nine browser only names, all of them in the sec- family, and three client only names that no browser has ever sent. Copying the browser sequence into an OrderedDict on the session closed the gap, and the 403 stopped.

One caution on that diff. A browser also emits requests for images, fonts and its own service worker, so the first flow tagged browser might be a favicon. Filter to the document request before you trust the comparison, which ~t html & ~m GET handles.

Once the two header lists agree, one variable is left in the experiment: whatever the path adds after the interceptor releases the request. I settle that by repeating both captures against an echo route through the anonymous pool I dial through and confirming the two arriving requests are byte for byte the same. When they match and the target still splits its verdict, the cause sits in the handshake or in the pace, and the header work is finished.

Exporting a flow to curl and reproducing it

Once a capture holds the request that failed, the fastest next step is a copy of it you can edit by hand. mitmproxy exports a selected flow directly.

In the terminal interface, move to the flow, press E, and pick curl from the list. From the command prompt inside the same interface:

:export.file curl @focus ~/repro.sh
:export.clip curl @focus
:export.file httpie @focus ~/repro-httpie.sh

@focus means the highlighted flow, @shown means everything the current filter matched, and @marked covers flows you tagged with m. Exporting @shown after filtering to ~c 403 gives a script per refusal in one keystroke.

The result is a full curl line with every header the client actually sent, in the order it sent them:

curl -X GET 'https://target.example/c/tools' \
  -H 'sec-ch-ua: "Chromium";v="126", "Not.A/Brand";v="8"' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'accept-encoding: gzip, deflate, br' \
  -H 'cookie: PHPSESSID=7f1c0ab93de241; cf_clearance=IZm9x4...' \
  --compressed

From there the work is subtraction. Delete one header, run it, see whether the code changes. I add -x with a working endpoint so the repro travels the same path as the original, and the two shapes the panel hands out both drop straight into that flag, IP:PORT and IP:PORT:LOGIN:PASS. For anything where the header block itself should stay between me and the origin, I run the repro over an HTTPS endpoint from the panel list, since the exchange rides inside a tunnel and the middle hop sees the host name alone.

Replay is the other half. mitmdump --replay-server run.flows answers from the capture with no network access, which lets you test a parser against real responses forever. -C flow.file replays your side of the exchange at the live target, useful for confirming that a fix survives a second attempt.

When mitmproxy is the thing that broke

Not every failure in a capture belongs to the target. Four of them belong to the interceptor, and telling them apart saves an afternoon.

Two behaviours are worth internalising. mitmproxy terminates TLS, so the handshake fingerprint the origin scores belongs to the interceptor, and a target refusing on handshake shape will behave differently while you watch. Confirm every finding by running once through the endpoint alone, with the interceptor removed. And keep the interceptor bound to 127.0.0.1: an interceptor listening on all interfaces is an open relay with a trusted certificate authority behind it.

For steady work I keep the capture rig on the same package the jobs use, because the point of the exercise is a chain that matches production. Private endpoints reserved for service clients hold still for the length of a session, which is what a session cookie needs, and server grade exits from the shared pool cover the volume side when a capture graduates into a full run. Package limits are worth checking before that scaling step, since flow limits divide across the addresses bound to an account.

The habit that came out of all this is small: capture first, theorise second. A refusal that took me a week the first time now takes eleven minutes, and the file it produces is evidence I can hand to somebody else. Once a capture names the party that refused you, the guide to reading block headers covers what each response family proves, the pool health checker walkthrough turns those findings into a probe that runs on a schedule, and the CI runner setup notes explain how to keep the same capture rig alive inside a build pipeline. When a status code needs pinning to a party before you open a flow at all, the response code reference sorts which codes come from an edge, which from a limiter and which from the application itself.