Golang HTTP proxy transport: from one Transport to a collector that runs all night

Go hands you a proxied request in three lines and then leaves you alone with everything that follows. My first version was exactly that: parse a URL, set Proxy, call Get. It worked on one page. Then I pointed the same code at 60 000 product URLs with 64 goroutines and watched the shape of the problem change completely. Handshakes repeated on almost every call, one silent target held a goroutine for eleven minutes, and the log filled with proxyconnect tcp lines that all looked identical while meaning four different things.

This guide walks the path my own collectors took. Transport comes first, because the proxy lives there and in no other place. Then the proxy function that picks an address per request, SOCKS5 through the dialer, pool sizing against goroutine count, the two deadlines, a retry wrapper with a hard ceiling, Colly on the same transport, and at the end httptrace plus the error strings. Numbers below came off my own runs against a pool of private server addresses where rotation happens inside the pool.

Transport carries the proxy, the Client only carries the deadline

The split between http.Client and http.Transport confuses people for about a week and then never again. The Client owns redirect policy, the cookie jar and one total deadline. The Transport owns connections: dialing, TLS, keep-alive, HTTP/2 and the proxy. Anything to do with a proxy address belongs on the Transport.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func main() {
	p, err := url.Parse("http://LOGIN:PASS@203.0.113.24:8000")
	if err != nil {
		panic(err)
	}

	tr := &http.Transport{
		Proxy:                 http.ProxyURL(p),
		MaxIdleConns:          128,
		MaxIdleConnsPerHost:   32,
		IdleConnTimeout:       75 * time.Second,
		TLSHandshakeTimeout:   8 * time.Second,
		ResponseHeaderTimeout: 20 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
	}

	c := &http.Client{Transport: tr, Timeout: 45 * time.Second}

	resp, err := c.Get("https://api.ipify.org")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	b, _ := io.ReadAll(resp.Body)
	fmt.Println(resp.StatusCode, string(b))
}

Three things in that block deserve attention before anything else.

http.ProxyURL(p) is a helper that returns a function always answering with the same address. The field itself has the signature func(*http.Request) (*url.URL, error), which is the hook the next section uses. Credentials written into the URL userinfo become a Proxy-Authorization header, and Go builds it for you on the CONNECT request.

The scheme in the proxy URL decides the transport to the endpoint. http:// means a plain proxy request for HTTP targets and a CONNECT tunnel for HTTPS targets. https:// means the hop to your endpoint is itself wrapped in TLS, which very few endpoints expect, and getting this wrong produces http: server gave HTTP response to HTTPS client on the first call. socks5:// in this field is accepted by Go and routes through the dialer, though the version with authentication support I trust lives in x/net/proxy and gets its own section below.

Build one Transport for the whole process and share it. Every &http.Transport{} created inside a function is a fresh connection pool with nothing in it, and a program that builds one per request pays a full handshake sequence per request while leaking descriptors. Mine lives in a package level variable built once in init. To drop the sockets on purpose, call tr.CloseIdleConnections().

One trap that never announces itself: http.DefaultTransport reads HTTP_PROXY, HTTPS_PROXY and NO_PROXY through http.ProxyFromEnvironment, and the environment is read once per process and cached. A shell variable someone exported months ago silently reroutes every call in a binary that thought it had no proxy at all. Setting Proxy: nil on your own Transport disables that path completely, and setting Proxy: http.ProxyURL(p) overrides it. Both forms are explicit and both survive a move from a laptop to a runner. The pool I point these at is described on the page covering private server proxies for Go collectors, where the address list arrives as IP:PORT and IP:PORT:LOGIN:PASS, the two shapes that paste directly into url.Parse.

Picking the address per request with a proxy function

The Proxy field being a function is the part of net/http I use most. It runs before every request that has no free connection waiting, it receives the outgoing *http.Request, and whatever it returns decides the route for that call. Returning nil, nil sends the request out directly with no proxy at all.

type picker struct {
	list []*url.URL
	n    uint64
	skip map[string]bool
}

func (p *picker) pick(r *http.Request) (*url.URL, error) {
	if p.skip[r.URL.Hostname()] {
		return nil, nil
	}
	i := atomic.AddUint64(&p.n, 1)
	return p.list[int(i%uint64(len(p.list)))], nil
}

tr := &http.Transport{Proxy: (&picker{
	list: parseAll(endpoints),
	skip: map[string]bool{"169.254.169.254": true, "localhost": true},
}).pick}

That skip map earns its place on any cloud runner. An instance metadata call sent through a remote endpoint hangs, times out, then produces an error message pointing nowhere near the cause. I lost most of an evening before adding those four lines.

Now the part that surprises everybody the first time. Go keys its idle connection cache by the combination of proxy address, target scheme and target host. Rotate across 8 endpoints in the proxy function and you have created 8 separate connection pools for a single target host, each holding its own share of MaxIdleConnsPerHost. Keep-alive reuse drops accordingly, and the run gets slower while looking busier. My habit is to hold one endpoint per goroutine group for the length of a batch and let rotation inside the pool supply the variety, which costs nothing and keeps reuse high. The mechanics of that are on the page about rotation that happens inside the pool, and the practical consequence for Go is that a single stable endpoint string in the proxy function still gives you a wide spread of exits.

Keep the function cheap, because it runs on the hot path. A lock, a file read or a health probe inside pick adds that cost to every dial in the process. Mine does an atomic increment and a map lookup.

SOCKS5 through x/net/proxy and the DialContext wrapper

For SOCKS5 with a username and password I reach for golang.org/x/net/proxy. It gives a proxy.Dialer, and the modern form of that interface carries DialContext, which is the method the Transport needs so cancellation actually propagates into the dial.

import (
	"context"
	"errors"
	"net"
	"net/http"
	"time"

	xproxy "golang.org/x/net/proxy"
)

func socksTransport(addr, user, pass string) (*http.Transport, error) {
	base := &net.Dialer{
		Timeout:   6 * time.Second,
		KeepAlive: 30 * time.Second,
	}
	d, err := xproxy.SOCKS5("tcp", addr,
		&xproxy.Auth{User: user, Password: pass}, base)
	if err != nil {
		return nil, err
	}
	cd, ok := d.(xproxy.ContextDialer)
	if !ok {
		return nil, errors.New("socks dialer exposes no DialContext")
	}

	return &http.Transport{
		DialContext:         cd.DialContext,
		MaxIdleConns:        256,
		MaxIdleConnsPerHost: 64,
		IdleConnTimeout:     75 * time.Second,
		TLSHandshakeTimeout: 8 * time.Second,
		ForceAttemptHTTP2:   true,
	}, nil
}

Four notes on that function, each one a bug I have already paid for.

Leave Proxy unset when DialContext holds a SOCKS dialer. Filling both makes the Transport open a CONNECT tunnel over a socket that already went through SOCKS, and the error you get back describes neither hop.

The type assertion to ContextDialer is what keeps context.WithTimeout meaningful. Falling back to the plain Dial method means a cancelled context does nothing until the dial finishes on its own schedule, which on a filtered port is 130 seconds of a goroutine doing nothing.

ForceAttemptHTTP2 matters because Go stops negotiating HTTP/2 automatically as soon as you supply your own DialContext. Targets that speak h2 will happily fall back to HTTP/1.1 without telling you, and the throughput difference on a deep pagination job is visible in the run time.

Name resolution moves to the SOCKS server. The dialer sends the hostname in the request when the address is a name, so lookups happen at the endpoint. Two consequences follow: your local resolver never learns which hosts the run touched, and a no such host error in the log means the endpoint failed the lookup. Both forms of credential, the bound address and the login pair, come with a SOCKS5 package with login and password, and the login form is the one I use whenever the binary runs somewhere whose address moves.

MaxIdleConnsPerHost and IdleConnTimeout against the goroutine count

Here is the single most expensive default in net/http. MaxIdleConnsPerHost is 2. Two. Run 64 goroutines against one host and 62 of them finish a request, find the idle slot full, and close a perfectly healthy socket. The next call opens a new one. TCP handshake, TLS handshake, CONNECT round trip, every time.

Nothing warns you. Throughput sits at a third of what it should be, and the target sees a churn pattern that looks nothing like a browser session. I found it on a run whose profile showed 71 percent of wall time inside dial and handshake.

The fields that matter and how I size them:

MaxIdleConnsPerHost sits slightly above the goroutine count for single host jobs. Slightly, because sockets held open past the burst do nothing except occupy descriptors.

MaxIdleConns is the process wide ceiling across all hosts. On a job touching 40 domains it needs real headroom or the cache evicts live sockets belonging to hosts you are still working on.

MaxConnsPerHost caps total connections per host, active plus idle, and blocks a goroutine when the cap is reached. Left at 0 it is unlimited. I set it on wide runs as a brake I can see, since a blocked goroutine is visible in the timings while silent socket churn is invisible.

IdleConnTimeout decides how long a spare socket waits before Go closes it. Set it above what the target's own keep-alive allows and you inherit a stream of EOF errors from writing to connections the far side already dropped. Most targets sit between 60 and 75 seconds, so I stay under that.

Read the last two rows against the first four and the shape of the trade appears. Narrow runs push per host reuse as high as it goes. Wide runs spread a fixed descriptor budget across many hosts, so per host numbers come down while the process wide ceiling stays flat. Below both sits a limit that has nothing to do with Go: ulimit -n. A run at 800 goroutines with generous idle caps will hit socket: too many open files on a default Linux configuration, and the error surfaces as random dial failures scattered across the log.

The other ceiling comes from the package. The terms I run allow 1000 parallel connections, the corporate one goes to 3000, and they do not stack: with two addresses bound the allowance splits in half between them. Setting a goroutine count above that produces queued work that reads exactly like a slow target. Traffic volume plays no part in the arithmetic here, since access is sold by period with the traffic unmetered. What each period allows in parallel is listed alongside server addresses that hold long keep-alive sessions, and that figure is the only external number the table above depends on.

One more habit, cheap and worth it. Drain the body before closing it, always:

func drain(resp *http.Response) {
	io.Copy(io.Discard, resp.Body)
	resp.Body.Close()
}

A body closed with bytes still unread cannot go back into the idle pool. Go closes the socket. I have seen this single omission cut reuse to nothing on a job that had every pool number tuned correctly.

Two deadlines: the Client timeout and the request context

http.Client.Timeout covers everything: dial, handshakes, request write, response headers, and the full body read. It is a wall clock ceiling on the whole exchange. That makes it a blunt instrument, and it is still the one guard you should never omit, because a Client with no timeout will hold a goroutine forever against a target that accepts the connection and goes quiet.

The context deadline works at a different level and composes better.

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
	return err
}
req.Header.Set("Accept-Encoding", "gzip")

resp, err := client.Do(req)
if err != nil {
	return err
}
defer drain(resp)

Whichever fires first wins. My arrangement uses a generous Client.Timeout as a backstop and a per URL context tuned to the target, which lets one slow section of a site get 40 seconds while the rest of the run stays on 15.

Two details cost people afternoons. Missing defer cancel() leaks the timer and the goroutine watching it, which on a long run reads as memory that climbs and never comes back. The context deadline also keeps running while you read the body, so a large file under a short context gets cancelled halfway down with no obvious cause in the message.

Between those two sit the Transport timeouts from the earlier table. ResponseHeaderTimeout is the one I reach for when a target stalls: it caps the wait for the first response header without touching the body read, so a slow generating page is caught early while a legitimate 200 MB download runs to completion.

Retries with a ceiling, and the codes that must never repeat

Go ships no retry logic. Writing it as a RoundTripper puts the loop below your application code, which means every client in the process gets it for free and no caller has to remember anything.

type retrier struct {
	base    http.RoundTripper
	max     int
	backoff time.Duration
}

var repeatable = map[int]bool{
	408: true, 425: true, 429: true,
	500: true, 502: true, 503: true, 504: true,
}

func (rt *retrier) RoundTrip(req *http.Request) (*http.Response, error) {
	req = req.Clone(req.Context())

	for attempt := 0; ; attempt++ {
		resp, err := rt.base.RoundTrip(req)

		if err == nil && !repeatable[resp.StatusCode] {
			return resp, nil
		}
		if attempt >= rt.max {
			return resp, err
		}
		if err != nil && !temporary(err) {
			return nil, err
		}
		if err == nil {
			wait := rt.backoff << attempt
			if s := resp.Header.Get("Retry-After"); s != "" {
				if n, e := strconv.Atoi(s); e == nil {
					wait = time.Duration(n) * time.Second
				}
			}
			drain(resp)
			if !sleepCtx(req.Context(), wait) {
				return nil, req.Context().Err()
			}
			continue
		}
		if req.GetBody != nil {
			b, e := req.GetBody()
			if e != nil {
				return nil, e
			}
			req.Body = b
		}
		if !sleepCtx(req.Context(), rt.backoff<<attempt) {
			return nil, req.Context().Err()
		}
	}
}

Four rules live in that function. The request gets cloned, because the RoundTripper contract forbids modifying the one you were handed. The body is rewound through req.GetBody, which http.NewRequest populates for you on byte slices, strings and buffers, and which is nil for an arbitrary io.Reader you supplied yourself. The sleep respects the context, so a cancelled batch stops within milliseconds. And the previous response is drained before the next attempt, keeping the socket eligible for reuse.

Worst case duration is arithmetic worth doing once, because it explains slow runs better than a profiler will. With max=3, a backoff of 500 ms and a 25 second context, one URL can occupy a goroutine for the full 25 seconds and produce nothing. At 48 goroutines and a 3 percent failure rate over 60 000 URLs, that is roughly 15 minutes of pure waiting folded into the run.

The status list is where most retry code goes wrong.

Two of those rows carry the run. A 403 repeated four times turns a modest refusal rate into thousands of identical refusals inside a few minutes, and every attempt adds another sample to whatever is scoring you. I log Server, Cf-Ray, X-Cache and Set-Cookie headers on every 403, push the URL to a slow queue, and pick it up later on a fresh exit.

A 407 arrives from your own endpoint. In Go the error text is the CONNECT status line, so you see Proxy Authentication Required sitting in a Get "https://..." wrapper. Three causes cover nearly all of them: a credential pair with a pasted trailing space, a machine whose address is absent from the bound list, or a period of access that has ended. Repeating it wastes a goroutine slot per attempt for the same answer, so my handler cancels the batch context immediately. A 407 on one worker means a 407 on every worker. Binding takes seconds in the panel, packages come with two bindable addresses you can swap freely, and the login form arrives with an HTTPS endpoint that carries the credential pair, which keeps the first hop encrypted while the pair travels.

Colly on top of the same transport

Colly is a collector framework with a callback model, and underneath it there is an ordinary http.Client with an ordinary Transport. Everything above still applies. The trick is putting the pieces together in the right order.

import (
	"github.com/gocolly/colly/v2"
	cproxy "github.com/gocolly/colly/v2/proxy"
)

c := colly.NewCollector(
	colly.Async(true),
	colly.AllowedDomains("shop.example.com"),
	colly.MaxDepth(4),
	colly.UserAgent(uaString),
)

c.WithTransport(tr)          // first: our sized transport

sw, err := cproxy.RoundRobinProxySwitcher(
	"http://LOGIN:PASS@203.0.113.24:8000",
	"http://LOGIN:PASS@203.0.113.25:8000",
)
if err != nil {
	log.Fatal(err)
}
c.SetProxyFunc(sw)           // second: it edits tr.Proxy in place

c.SetRequestTimeout(30 * time.Second)

SetProxyFunc takes a colly.ProxyFunc, whose signature is the same func(*http.Request) (*url.URL, error) from the second section. Internally it type asserts the current transport to *http.Transport and writes into its Proxy field. Call WithTransport afterwards and you throw the proxy function away along with the transport, and the run goes out through your own address with no error anywhere. Order matters. I check it with a first request to an address reflection endpoint before anything else runs.

Watch the import naming. golang.org/x/net/proxy and github.com/gocolly/colly/v2/proxy both want the identifier proxy, so a file using SOCKS5 with Colly needs an alias on at least one. I alias both, always.

Reading which endpoint served a response is a one liner, and it is the measurement that makes everything else debuggable:

c.OnResponse(func(r *colly.Response) {
	log.Printf("%d %s via %s %dB",
		r.StatusCode, r.Request.URL, r.Request.ProxyURL, len(r.Body))
})

c.OnError(func(r *colly.Response, err error) {
	log.Printf("ERR %s via %s: %v",
		r.Request.URL, r.Request.ProxyURL, err)
})

r.Request.ProxyURL holds the endpoint Colly actually used for that call. Counting distinct exits per thousand responses gives me the spread of the run, and on a pool of roughly 12 000 active addresses across more than 200 countries that number climbs fast. A spread that stays flat while the run grows tells me a proxy function is caching something it should not.

Holding Colly parallelism under the package thread limit

Colly's Async(true) alone gives you unbounded concurrency, which is almost never what you want. LimitRule is the brake, and it applies per matching domain.

c.Limit(&colly.LimitRule{
	DomainGlob:  "*.example.com",
	Parallelism: 24,
	Delay:       150 * time.Millisecond,
	RandomDelay: 350 * time.Millisecond,
})

c.OnHTML("a.product[href]", func(e *colly.HTMLElement) {
	e.Request.Visit(e.Attr("href"))
})

c.Visit("https://shop.example.com/catalog")
c.Wait()

Parallelism is enforced by a weighted semaphore held per domain, so a run against 6 domains with DomainGlob: "*" and Parallelism: 24 reaches 144 concurrent requests at peak. That arithmetic catches people out constantly. Either write one rule per domain with its own number, or divide the global figure by the number of domains you expect.

Delay is a fixed pause after each request on that domain, and RandomDelay adds a uniform random extra on top. Both are applied per worker slot, so the effective request rate is roughly Parallelism divided by the mean delay. With the numbers above that comes to about 73 requests per second against one domain, which is more than most targets tolerate. Calculate that figure before the run starts. Reading it off a graph afterwards costs you the batch.

The ceiling from your package sits above all of this. Parallelism multiplied by the number of matched domains has to land under the parallel connection allowance, and remember the halving when two addresses are bound. Working out how many workers a given catalogue depth needs against that figure is arithmetic I stopped doing by hand, and the calculator linked at the end of this guide turns page count, page weight and the thread ceiling into a number you can paste into Parallelism. The same arithmetic sits behind the thread setting A-Parser exposes for a pool, and the allowance itself is listed with each access period.

Two more Colly settings go on every long job. c.SetRequestTimeout writes into the underlying Client.Timeout, and it has to sit above the sum of your retry attempts or the wrapper never finishes its second try. colly.CacheDir writes responses to disk, so a re-run after a parser change costs no requests.

httptrace: splitting a request into phases

When a run is slower than it should be, the useful question is which phase is eating the time. net/http/httptrace answers it with callbacks fired at each stage, and it costs almost nothing to leave switched on for a sample of requests.

func timed(ctx context.Context) (context.Context, func() string) {
	var start, connDone, tlsDone, first time.Time
	var reused, wasIdle bool
	start = time.Now()

	t := &httptrace.ClientTrace{
		GotConn: func(i httptrace.GotConnInfo) {
			reused, wasIdle = i.Reused, i.WasIdle
		},
		ConnectDone: func(_, _ string, _ error) {
			connDone = time.Now()
		},
		TLSHandshakeDone: func(tls.ConnectionState, error) {
			tlsDone = time.Now()
		},
		GotFirstResponseByte: func() {
			first = time.Now()
		},
	}

	report := func() string {
		return fmt.Sprintf("reused=%v idle=%v tcp=%v tls=%v ttfb=%v",
			reused, wasIdle,
			connDone.Sub(start), tlsDone.Sub(connDone), first.Sub(start))
	}
	return httptrace.WithClientTrace(ctx, t), report
}

GotConn is the field I read first on every investigation. Reused false on most calls means keep-alive is broken somewhere, and the cause is one of four things: MaxIdleConnsPerHost too low, bodies closed without draining, a proxy function rotating endpoints on every call, or IdleConnTimeout above what the target keeps open.

With a proxy in the picture the phase names shift meaning, and this catches everyone once. ConnectStart and ConnectDone describe the TCP connection to your endpoint. The TLSHandshake pair describes the handshake with the final target, performed through the tunnel after CONNECT succeeds. DNS callbacks report the lookup of the endpoint hostname and stay silent when it is written as an IP address.

A trace from one of my own runs, single call to a catalogue page, cold connection: TCP to the endpoint 41 ms, tunnel setup 44 ms, TLS with the target 96 ms, first byte at 388 ms. The same URL on a warm connection: first byte at 112 ms. That gap of 276 ms multiplied across 60 000 URLs is four and a half hours of handshakes, which is the whole argument for the pool sizing table above, stated as a measurement.

Go error strings, line by line

Error text in Go is wrapped by every layer it passes through, so the message you read carries the whole route. Learning to read them backwards, innermost cause first, is the skill that saves the most time.

Three rows need a sentence more than a table cell allows.

proxyconnect tcp is the prefix Go puts on any failure while establishing the tunnel to your endpoint. Everything after it describes the endpoint hop. When someone reports a site blocking them and pastes one of these, the site has seen no packet at all yet.

EOF with no other text is the classic idle keep-alive race. Go picked a socket out of the idle pool, the far side had already closed it, and the write landed on a dead connection. It is harmless at a low steady rate and retryable in one attempt. A rate that climbs through the night means IdleConnTimeout is above what the target allows, and the fix is a smaller number there. A bigger retry budget leaves the race in place.

connection reset by peer on a rotating pool appears at a low constant rate on any long run, because a socket that was fine a second ago belongs to an exit that has moved on. I count these per hour and treat the count as a health number for the job. Flat is normal operation. Climbing means the goroutine count outgrew what the target tolerates, and the answer is a smaller Parallelism with a slightly larger RandomDelay.

The same shapes translate across ecosystems, and the neighbouring guides carry them: the agent and pool arithmetic written in JavaScript sits in the Node HTTP client guide for axios, got and undici, the framework equivalent of the Colly section with its own concurrency model is the Scrapy downloader middleware guide, and the same options expressed as handles and request objects appear in the PHP curl and Guzzle proxy guide. Before you set Parallelism on a new target, push the depth and the thread ceiling through the pool and thread calculator and start from a number that already fits.