curl through a proxy: the flags that decide what comes back

Almost every proxy question I have chased this year was answered by one curl command typed into a terminal. The command itself is short. Reading its output correctly is the skill, and that skill lives inside about a dozen flags that most people copy from a forum post and never look at again.

My nightly job pulls 312 listing pages from 4 retail domains, and before it starts, a verification pass walks 40 endpoints from the current batch and grades them. Both pieces are built on curl. When the job goes quiet at three in the morning, my debugging session is a terminal, one endpoint and the flags below, in roughly the order they appear here: address forms, credentials, name resolution, headers, phase timings, the handshake, retries, encoding, environment, cookies, certificates.

Each flag answers one narrow question. Knowing which question belongs to which flag is the difference between a measurement and a guess, and I have wasted whole evenings rotating perfectly healthy addresses because I asked the wrong flag.

The -x flag and the shapes an address can take

-x attaches a proxy to a single call. --proxy is the same option spelled out, and I use the long form in scripts because six months later nobody remembers what -x was.

curl -x http://93.184.16.44:8080 -s -o /dev/null -w '%{http_code}\n' https://example.net/catalog

What trips people is how much curl fills in when you leave something out. Give it a bare host with no scheme and it assumes HTTP. Give it a scheme with no port and it assumes 1080, which is the SOCKS port, so an HTTP endpoint written without a port fails with a connection error that looks like a dead address. I write the scheme and the port every time.

That fourth row confuses people constantly. http:// in the -x argument describes the connection between curl and the endpoint, and it says nothing about the target. A plain HTTP endpoint still carries your HTTPS traffic perfectly well: curl opens a CONNECT tunnel and the TLS session runs end to end with the target inside it. https:// in the -x argument means something narrower, that the first hop itself is wrapped in TLS, so anyone watching the wire between your machine and the endpoint sees an encrypted stream with no target hostname in the clear. On a laptop sitting in a coworking network I want that first hop encrypted, which is why the batch I keep for travel comes from HTTPS endpoints in my own list with the port that terminates TLS at the exit.

The --proxy-user flag against the login written into the address

Credentials can travel two ways, and both work on the first attempt, which is why the wrong one spreads.

curl -x http://A19f42:x7Qd21ka@93.184.16.44:8080 https://example.net/

curl -x http://93.184.16.44:8080 --proxy-user A19f42:x7Qd21ka https://example.net/

The first line buries the pair inside the address string. The second gives it a flag of its own. The second form is the one I ship, for reasons that have nothing to do with curl parsing. A password sitting inside a URL gets copied into shell history, into CI job logs that keep the full command line, and into any monitoring agent that samples the process table. A password in --proxy-user lands in exactly the same places, so on its own it fixes nothing. The gain arrives when you move the flag out of the command entirely:

Written into ~/.curlrc, or into a file handed over with --config and locked to mode 600, the pair looks like this:

proxy = "http://93.184.16.44:8080"
proxy-user = "A19f42:x7Qd21ka"

curl --config /etc/collector/proxy.conf -s -o /dev/null -w '%{http_code}\n' https://example.net/

Now the process table shows a path and no secret. There is a parsing detail that also pushes me toward the flag: characters like @, : and # inside a password break the URL form and have to be percent encoded, while --proxy-user takes the raw string. A password with an @ in it, pasted into the URL form, produces Could not resolve proxy with a hostname made of the password tail, and the message points at DNS while the real cause sits three characters earlier.

The authentication scheme has its own flags. --proxy-basic is the default, --proxy-digest and --proxy-ntlm cover corporate gateways, and --proxy-anyauth asks curl to look at the Proxy-Authenticate header and pick. I set --proxy-basic explicitly on scripted calls, since --proxy-anyauth costs an extra round trip on every request while it negotiates.

The path with no credentials at all is IP binding, and it is what my production nodes use. I register the machine's own address in the dashboard, and after that the endpoint answers that machine with an empty auth header. My package carries two bindable addresses, which covers a production node and a staging node, and the list arrives in both shapes, IP:PORT for bound machines and IP:PORT:LOGIN:PASS for anything else. Long running jobs sit on private server addresses bound to my machines with no secret anywhere in the repository.

socks5 against socks5h and the path a hostname travels

One letter separates these two schemes, and that letter decides which machine performs the DNS lookup.

With socks5://, curl resolves the target hostname on your own machine and sends the resulting IP address to the endpoint. With socks5h://, curl sends the hostname itself and the endpoint performs the lookup. The older spellings --socks5 and --socks5-hostname map to the same pair of behaviours.

Two things follow from that, and both matter more than the syntax. The first is correctness. Any hostname that resolves differently at the exit, or fails to resolve at all from your network, works under one scheme and dies under the other. I have watched one target answer from a regional edge node under local resolution and from a different origin under remote resolution, with different headers, in the same minute.

The second is exposure. Local resolution writes every target domain into your own resolver's query log and into your network provider's. The tunnel carries the request bodies and leaves the list of who you are talking to in plain view outside it. For collection work that alone settles the question.

curl -x socks5h://93.184.16.60:1080 -s --max-time 15 https://ifconfig.co/json

curl -x socks5://93.184.16.60:1080 -s --max-time 15 https://ifconfig.co/json

The top line hands the hostname to the exit. The bottom line resolves it on my machine and hands over an IP. Running both against an address echo and comparing the two answers is the first check I make on any new endpoint. Identical bodies mean the exit is consistent under either path. A difference means something between us is doing hostname based routing, and I want to know that before the endpoint enters a batch. My hostname heavy jobs run on the SOCKS5 addresses I keep for remote lookups with socks5h written into the config file so nobody can forget the letter.

The -i flag and the full set of headers that comes back

-i prints response headers ahead of the body. -I sends a HEAD request, which filtering layers treat differently, so I stopped using it for diagnosis. The pattern I actually run writes headers to stdout and throws the body away:

curl -x http://93.184.16.44:8080 -s -o /dev/null -D - --max-time 20 https://example.net/catalog

Through a CONNECT tunnel, -i shows you the target's headers only. The endpoint's own response to the CONNECT request lives one layer below and appears under -v, which is the next section. Knowing that split saves confusion: a 407 never shows up in -i output, because the tunnel that would have carried the body was refused before the target was ever contacted.

The fields I read on every failing request, in order: via and x-forwarded-for tell me whether anything on the path announced my presence, x-cache and age tell me a CDN answered from storage, retry-after and x-ratelimit-remaining give me the target's own pacing signal, set-cookie shows whether a session was issued, cf-ray and server identify the filtering layer, and content-length against the visible body length catches truncation. A response that carries x-forwarded-for with my own machine's address is the loudest failure of all, and it means the endpoint added the header on the way out. I check for that on every new batch, which is why my checker runs against addresses that keep the forwarded header set quiet and flags anything that answers otherwise.

--proxy-header is the matching flag on the request side. Headers passed with -H go to the target inside the tunnel, headers passed with --proxy-header go to the endpoint on the CONNECT request itself. Sending an authentication header with -H to an endpoint that expects it on CONNECT produces a 407 and a very confused half hour.

The --write-out flag and the timing split that names the slow part

This is the flag that changed how I diagnose speed. --write-out prints variables after the transfer finishes, and the timing family gives you the request broken into phases.

curl -x http://93.184.16.44:8080 -s -o /dev/null \
  -w 'dns=%{time_namelookup} conn=%{time_connect} tls=%{time_appconnect} pre=%{time_pretransfer} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code} ip=%{remote_ip} bytes=%{size_download}\n' \
  --max-time 30 https://example.net/catalog

Every timing value is cumulative from the start of the call, so the useful numbers are the differences between neighbours.

Here is how I separate a slow node from a slow site with those four differences. Run the same target three times through the endpoint and once directly. If time_connect minus time_namelookup is consistently high across every target you try, the hop to the node is the cost, and no target change will help. If time_starttransfer minus time_pretransfer is high on one target and normal on the others through the same endpoint, the target is thinking, and the endpoint is doing its job. If both differences are normal while time_total runs long, the body is large or the return path is congested, and size_download divided by that last difference gives you the throughput figure to prove it.

On my nightly job the readings sit around conn at 0.09, tls at 0.21, ttfb at 0.34 and total at 0.61 seconds on a healthy night. When ttfb alone climbs past 1.5 while conn stays at 0.09, I stop looking at addresses and start looking at request pace, because the target is reacting to volume. Sustained collection at that pace runs on a pool built for page work at volume where rotation inside the pool happens without me maintaining a list.

For machine readable output, -w '%{json}' prints every variable curl knows as one object, and I append that line to a per batch journal file. After a month the journal answers questions no single run can: which hour of day carries the worst first byte times, whether failures cluster by endpoint or by target, and how much variance a given target shows before it starts refusing.

The -v flag and the CONNECT lines worth reading

-v writes the protocol conversation to stderr, and through a proxy it is the only place the tunnel negotiation is visible.

* Connected to 93.184.16.44 (93.184.16.44) port 8080
* allocate connect buffer
* Establish HTTP proxy tunnel to example.net:443
> CONNECT example.net:443 HTTP/1.1
> Host: example.net:443
> Proxy-Connection: Keep-Alive
<
< HTTP/1.1 200 Connection established
<
* CONNECT phase completed
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=example.net
*  SSL certificate verify ok.
> GET /catalog HTTP/2
< HTTP/2 200

Four lines in that block carry the diagnosis. Connected to proves the TCP hop to the node succeeded. HTTP/1.1 200 Connection established proves the endpoint agreed to open a tunnel to that host and port. SSL certificate verify ok proves the target's chain validated inside the tunnel. The final status line belongs to the target and to nobody else.

Now read the failure variants against that. HTTP/1.1 407 Proxy Authentication Required on the CONNECT response means your credentials were refused, the address itself is healthy, and rotating it wastes a working endpoint. HTTP/1.1 403 Forbidden on the CONNECT response means the endpoint declined that destination, which is a filtering rule at the node. A 403 that arrives after Connection established came from the target and has nothing to do with the endpoint. curl: (56) Proxy CONNECT aborted means the tunnel died mid negotiation, and that one I retry once before touching anything.

SOCKS endpoints print a different sequence, with SOCKS5 communication to example.net:443 followed by SOCKS5 request granted. If you see SOCKS5 server does not support user pass authentication the endpoint wants no credentials, and the fix is to bind the machine and drop the login from the string.

Two companions make -v usable at scale. --trace-ascii trace.txt writes the whole exchange with request bodies included, and --trace-time prefixes every line with a timestamp, which turns the trace into its own phase measurement. On machines I share with other people, the encrypted first hop from an HTTPS proxy package on my account keeps that CONNECT line unreadable to anyone capturing the local segment, and the trace still shows it to me in full.

The --retry flag and the conditions it fires on

--retry looks like a blanket safety net and it is deliberately narrower than that.

curl -x http://93.184.16.44:8080 \
  --retry 4 --retry-delay 2 --retry-max-time 60 \
  --connect-timeout 8 --max-time 45 \
  -s -o page.html -w '%{http_code} %{time_total}\n' https://example.net/catalog

Out of the box curl retries a transient set: transfer timeouts, and the status codes 408, 429, 500, 502, 503 and 504. A refused connection does not qualify, which surprises people, and --retry-connrefused adds it. --retry-all-errors widens the policy to everything including a mistyped flag, so a broken command runs five times and takes five times as long to tell you it was broken.

The pacing flags interact in a way worth stating plainly. --retry-delay fixes the pause between attempts, and without it curl doubles the wait each round starting from one second. --retry-max-time caps the total elapsed time across all attempts, measured from the first one. --max-time caps a single attempt. A job with --max-time 45 and --retry 4 and no --retry-max-time can occupy three minutes on one page, which is how a nightly run that used to finish by four ends up still going at six.

Since curl learned to honour Retry-After, a 429 carrying that header pushes the next attempt out to whatever the target asked for. I keep that behaviour on, because a target that names its own pace is telling me exactly how to stay welcome. My retry policy on collection jobs is four attempts, two second delay, sixty second ceiling, and a rule that anything failing all four gets written to a dead letter file with its -w line attached. That file is where the patterns show up, since the same target host repeated forty times means a target problem and forty different hosts through one endpoint means the node.

The --compressed flag and the encoding swap underneath it

--compressed sends an Accept-Encoding header listing whatever codecs your curl build supports, then decodes the response before you see it. Check what your build actually offers with curl -V and read the Features line for brotli and zstd.

curl -x http://93.184.16.44:8080 --compressed \
  -s -o /dev/null -w 'wire=%{size_download} code=%{http_code}\n' https://example.net/catalog

Two consequences follow, and the second one has bitten my extraction step more than once.

The first is what the site sends you. A page that arrives at 310 KB uncompressed arrives at roughly 44 KB with gzip active, and across 312 pages a night that difference is real bandwidth on the return hop. Access on my packages carries no ceiling on transfer, so the reason I keep compression on is latency: fewer bytes on the wire means a shorter gap between time_starttransfer and time_total, and on a long return path that gap dominates.

The second is what --write-out reports. size_download counts bytes received on the wire, so with compression active it reports the compressed figure. A monitoring check that alarms when a page drops below 100 KB will fire on every single request the moment somebody adds --compressed to the command, and the page never changed at all.

The failure mode people hit is setting the header by hand:

curl -x http://93.184.16.44:8080 -H 'Accept-Encoding: gzip' https://example.net/catalog

-H announces that you accept gzip. It does not enable decoding. The target obliges, the body arrives compressed, and your terminal fills with binary. Let curl own both halves with --compressed and the problem disappears. The opposite flag is --raw, which disables all content decoding including transfer encoding, and it belongs in exactly one situation, when you are saving the response to compare bytes against a reference capture.

Through a proxy there is a third party in this negotiation. Some intermediaries recompress or strip the encoding on the way back, so if a body arrives undecoded despite --compressed, read the content-encoding header from -D - and see what the target claimed to send. A mismatch between the claim and the reality is a path problem worth a note in the journal.

Environment variables, NO_PROXY and cookies carried between requests

curl reads proxy settings from the environment when no -x is given, and this is the single most common source of a measurement that lies to you.

export http_proxy="http://93.184.16.44:8080"
export https_proxy="http://93.184.16.44:8080"
export NO_PROXY="localhost,127.0.0.1,.internal,collector.local"

The lowercase https_proxy is the one curl honours; the uppercase spelling of that particular variable is ignored on purpose, because CGI environments can be fed a HTTPS_PROXY header value by a remote caller. ALL_PROXY covers every scheme at once. -x on the command line beats all of them, and --noproxy '*' disables proxying for one call, which is how I take a direct baseline on a machine whose shell profile exports a proxy without telling me. That baseline is what the phase timings get compared against.

NO_PROXY takes a comma separated list of hosts with no scheme and no port. A leading dot matches subdomains, a bare hostname matches that host, and * on its own disables everything. IP ranges in CIDR form work in current builds. Any internal service your script also talks to belongs in that list, because a request to your own collector sent through a far endpoint fails in a way that points at your code.

Cookies are the other piece of state that has to survive between calls:

The first call logs in and writes the jar. Every call after it reads the jar and writes it back.

curl -x http://93.184.16.44:8080 -c jar.txt -s -o /dev/null \
  -d 'user=probe&pass=probe' https://example.net/session

curl -x http://93.184.16.44:8080 -b jar.txt -c jar.txt -s -L \
  -o listing.html https://example.net/catalog

-b reads, -c writes at the end of the transfer, and both together carry a session forward. Passing only -b means every cookie the server issues during the call is discarded, so a session that refreshes its token silently dies on the next request. -L follows redirects and keeps the jar across them, which matters because the cookie that authorises you is often set on the redirect and never on the final page.

One rule I keep without exception: a jar belongs to one exit. Reusing a jar across two different endpoints presents the same session token from two different addresses, and that pattern is trivial to spot at the target. Fresh exit means fresh jar, and my scripts name the jar after the endpoint so mixing them takes deliberate effort.

Certificate checks and the errors that live on your machine

curl: (60) SSL certificate problem: unable to get local issuer certificate is the message people meet first, and nine times out of ten it describes your machine. The chain the target presented is fine, and your CA bundle is old, missing, or being looked for in a path that does not exist. Point curl at a current bundle with --cacert /etc/ssl/certs/ca-certificates.crt or set CURL_CA_BUNDLE once for the whole environment, and the error goes away without touching the endpoint.

Two flags disable verification and they are not interchangeable. -k turns off verification of the target's certificate, which is the one people reach for and the one that hides real problems. --proxy-insecure turns off verification only for the TLS session between curl and an HTTPS endpoint, leaving the target's chain checked as normal. --proxy-cacert supplies a bundle for that first hop alone. On a self signed endpoint certificate the correct move is --proxy-cacert with the endpoint's own certificate file, and the target stays verified.

%{ssl_verify_result} in --write-out reports the verification code as a number, and it belongs in any automated checker. Zero means the chain validated. A non zero value with a 200 status tells you -k is somewhere in your command and you have been collecting unverified responses for a while.

The grouping in that column is what I care about. Codes 5, 60 and most 7 results are mine to fix. Codes 35, 52 and 56 sit at the target or on the path between the exit and the target. A 407 is a credential problem wearing a network problem's clothing, and it fools people into pulling healthy addresses out of rotation every day.

Two habits close the loop for me. Every failing call gets re run once with -v before I form an opinion, since the CONNECT block names the side that refused in two lines. And every endpoint that produces an unexplained failure gets a fresh measurement against an address echo with the full -w string, so the phase timings go into the journal next to the error text. Access on my package is a period of access to the pool, with automatic rotation inside it and roughly 12000 live addresses in the list, so a suspect exit costs me nothing to replace.

The neighbouring guides on this site pick up the same work in other clients: session objects, connection pooling and retry adapters in Python are covered in requests sessions and retries, the three Node HTTP clients and their differing proxy agents in axios, got and undici with a proxy, and the framework level version of everything above in writing a Scrapy proxy middleware. When a dashboard line has to become the exact -x argument, the config file entry and the --proxy-user pair without a transcription slip at midnight, the connection string parser splits the four part form apart and prints each client's version ready to paste.