Playwright proxy setup: every level where an address gets attached

Playwright hands you three places to attach a proxy, and the place you pick decides what the rest of the run is able to do. I have rewritten the same collection job four times for exactly that reason. The first version carried the address on launch(), so every tab in the process left through one exit. The second moved the setting down to the context, which let me hold eight sessions with eight different exits inside a single browser process. The remaining two rewrites were about credentials and about DNS, and those two swallowed more debugging hours than the rest of the project put together.

This walkthrough follows the levels in the order I set them up on a fresh machine: launch, context, credentials, scheme, interception, timeouts, container, concurrency, diagnostics. Every snippet appears twice, once for Node and once for Python, because our crawlers run in Node while the verification suite that grades them runs in Python. The numbers come from one job of mine: 46 product pages across three retail domains, repeated every night for a month, with the timings written into a small journal I keep per address batch.

Level one: the proxy that comes up with the browser

The launch option is the shortest path from zero to a working request. One object on launch(), and every context born in that browser inherits the exit.

const { chromium } = require('playwright');

const browser = await chromium.launch({
  headless: true,
  proxy: {
    server: 'http://93.184.16.44:8080',
    username: 'A19f42',
    password: 'x7Qd21ka',
  },
  args: ['--disable-blink-features=AutomationControlled'],
});

const page = await browser.newPage();
await page.goto('https://example.net/catalog', { waitUntil: 'domcontentloaded' });

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={
            "server": "http://93.184.16.44:8080",
            "username": "A19f42",
            "password": "x7Qd21ka",
        },
    )
    page = browser.new_page()
    page.goto("https://example.net/catalog", wait_until="domcontentloaded")

Two fields on that object get overlooked. server accepts a scheme, a host and a port, and the scheme carries meaning that I unpack further down. bypass takes a comma separated list of hosts that skip the tunnel completely, and it saves an afternoon on any job that also talks to a local API: bypass: 'localhost,127.0.0.1,*.internal'. Without it the browser tries to reach your own service through the far exit, the request dies somewhere on the way, and the stack trace points at your code.

I keep the launch level for one shot jobs and for smoke tests against a pool sized for collection runs, where a single exit for the whole process is the behaviour I want. Anything that holds more than one identity moves down a level.

Level two: a proxy per context, and why sessions need it

A BrowserContext is an isolated cookie jar with its own storage, cache and permissions. Two contexts in one browser share nothing at the application layer. Playwright lets each one carry its own proxy, which means one Chromium process can drive eight sessions leaving through eight addresses at the same time.

const ctxA = await browser.newContext({
  proxy: { server: 'http://93.184.16.51:8080', username: 'A19f42', password: 'x7Qd21ka' },
  locale: 'en-GB',
  viewport: { width: 1366, height: 768 },
});

const ctxB = await browser.newContext({
  proxy: { server: 'http://93.184.16.52:8080', username: 'A19f42', password: 'x7Qd21ka' },
  locale: 'de-DE',
});

const [pA, pB] = await Promise.all([ctxA.newPage(), ctxB.newPage()]);

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(proxy={"server": "http://per-context"})
        ctx_a = await browser.new_context(
            proxy={"server": "http://93.184.16.51:8080",
                   "username": "A19f42", "password": "x7Qd21ka"})
        ctx_b = await browser.new_context(
            proxy={"server": "http://93.184.16.52:8080",
                   "username": "A19f42", "password": "x7Qd21ka"})
        page_a, page_b = await ctx_a.new_page(), await ctx_b.new_page()

asyncio.run(main())

Look at the placeholder in the Python sample. Chromium builds before Playwright 1.29 required a proxy on the browser itself for the per context setting to take effect at all, and the value could be any string, since every context overrode it anyway. Newer builds drop the requirement. I still pass the placeholder in code that ships to machines whose Playwright version I do not control, because a missing placeholder produces the worst possible failure mode: the contexts work, the pages load, and every one of them leaves through the machine's own address.

The context level is what account work needs. A session that has to look like one visitor from the first request to the last belongs on one endpoint held for its whole life, the same pairing Dolphin Anty applies to a browser profile, with the context living exactly as long as the session does. When the session ends I close the context, and the storage goes with it.

Level three: getting the login and the password accepted

Credentials go in the username and password fields. They do not go in the server URL. Playwright parses server as a scheme, host and port, and a string shaped like http://user:pass@ip:port produces behaviour that changes between browser engines, which is a debugging session nobody needs.

const ctx = await browser.newContext({
  proxy: {
    server: 'http://93.184.16.51:8080',
    username: process.env.PROXY_USER,
    password: process.env.PROXY_PASS,
  },
});

import os

ctx = browser.new_context(proxy={
    "server": "http://93.184.16.51:8080",
    "username": os.environ["PROXY_USER"],
    "password": os.environ["PROXY_PASS"],
})

Now the part that catches people the first time they move from HTTP to SOCKS. Chromium carries no username and password authentication for SOCKS5 at all. The fields are accepted by the Playwright API, the browser drops them on the floor, and the connection either hangs or dies with a socks error that says nothing about credentials. Firefox handles SOCKS5 authentication, Chromium does not, and no launch argument changes that.

The way around it is IP binding. I register the crawler machine's own address in the dashboard, and from that moment the endpoint answers that machine with no credentials at all. My packages come with two bindable addresses, which covers a production node and a staging node, and the list arrives in two shapes: IP:PORT for bound machines, IP:PORT:LOGIN:PASS for everything else. For Chromium jobs I take the bound form of a SOCKS5 list from my own dashboard and hand Playwright a bare socks5://host:port string with no auth fields.

When credentials are genuinely wrong, Chromium surfaces net::ERR_INVALID_AUTH_CREDENTIALS on the failed request. That message means the tunnel opened and the far side refused the pair. A tunnel that never opened produces a different error, and telling those two apart saves you from rotating a perfectly good address.

Level four: socks5, socks5h, and where the domain name gets resolved

Here is the error that cost me the most time, and it hides behind one letter. The socks5 and socks5h schemes describe two different DNS paths, and the tools you use disagree about which name means which path.

In curl, socks5:// resolves the hostname on your machine and sends the resulting IP to the proxy. socks5h:// sends the hostname itself and lets the proxy resolve it. In Chromium there is no socks5h scheme at all: socks5:// already means remote resolution, and socks4:// is the one that resolves locally. Firefox keeps the choice in a preference, network.proxy.socks_remote_dns, which Playwright can set at launch.

const browser = await firefox.launch({
  proxy: { server: 'socks5://93.184.16.60:1080' },
  firefoxUserPrefs: {
    'network.proxy.socks_remote_dns': true,
    'media.peerconnection.enabled': false,
  },
});

browser = p.firefox.launch(
    proxy={"server": "socks5://93.184.16.60:1080"},
    firefox_user_prefs={
        "network.proxy.socks_remote_dns": True,
        "media.peerconnection.enabled": False,
    },
)

Why any of this matters in practice: a verification script that passes with curl on socks5:// and a crawler that fails on the same string in Playwright are doing two different things on the wire. Worse, local resolution puts the full list of your target domains into your own DNS logs and into your provider's, which turns a tunnel into a partially open window. I test every new endpoint with socks5h:// in curl first, then with socks5:// in Chromium, and I compare the two answers before the address enters a batch.

curl -x socks5h://93.184.16.60:1080 -s --max-time 15 https://ifconfig.co/json | head -c 200

There is a second leak on the same theme. WebRTC can hand a site your real address through an ICE candidate while every HTTP request goes through the tunnel correctly. The Firefox preference above closes it. On Chromium I disable the feature through --force-webrtc-ip-handling-policy=disable_non_proxied_udp in the launch arguments.

Level five: route interception and the requests you never wanted

Playwright's route handler sits between the page and the network, and it is the single biggest lever on how much traffic a run actually generates. A retail product page in my job pulled 2.6 MB across 84 requests. After filtering it pulled 380 KB across 19 requests, and the median page time dropped from 4.1 seconds to 1.3.

await context.route('**/*', (route) => {
  const type = route.request().resourceType();
  if (['image', 'media', 'font'].includes(type)) return route.abort();

  const url = route.request().url();
  if (/googletagmanager|doubleclick|hotjar|facebook\.net/.test(url)) return route.abort();

  return route.continue();
});

BLOCKED_TYPES = {"image", "media", "font"}
BLOCKED_HOSTS = ("googletagmanager", "doubleclick", "hotjar", "facebook.net")

def gate(route):
    req = route.request
    if req.resource_type in BLOCKED_TYPES:
        return route.abort()
    if any(h in req.url for h in BLOCKED_HOSTS):
        return route.abort()
    route.continue_()

context.route("**/*", gate)

Two warnings from my own journal of failed runs. Blocking stylesheet looks tempting and it broke three of my target sites, because their price block is rendered into a container whose size comes from CSS, and a zero height container makes the text unreadable to the extractor. Blocking xhr breaks more than that, since most catalogue prices arrive over exactly those calls.

The other warning concerns detection. A browser that requests HTML and scripts while never touching a single image produces a request pattern that stands out in any server log. On the domains that grade sessions I keep images on and cap them by size, aborting anything above 200 KB and letting the small ones through. The saving is smaller and the session survives longer.

Route handlers register per context, so each of your parallel sessions needs its own registration. A handler attached to page covers that page alone, which is the shape I use when one tab in a context must behave differently from its siblings.

Level six: timeouts and retries around a proxied navigation

Default timeouts assume a direct connection. Add a tunnel and every number needs revisiting, because a handshake through an endpoint in another region adds 200 to 400 ms before the first byte of the target response even starts moving.

context.setDefaultNavigationTimeout(45000);
context.setDefaultTimeout(20000);

async function loadWithRetry(context, url, attempts = 3) {
  for (let i = 1; i <= attempts; i++) {
    const page = await context.newPage();
    try {
      const res = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 45000 });
      if (res && res.status() < 400) return { page, res };
      await page.close();
    } catch (err) {
      await page.close();
      if (i === attempts) throw err;
      await new Promise(r => setTimeout(r, 1500 * i));
    }
  }
}

def load_with_retry(context, url, attempts=3):
    last = None
    for i in range(1, attempts + 1):
        page = context.new_page()
        try:
            res = page.goto(url, wait_until="domcontentloaded", timeout=45000)
            if res and res.status < 400:
                return page, res
            page.close()
        except Exception as err:
            last = err
            page.close()
            time.sleep(1.5 * i)
    raise last

The retry loop above is deliberate about one thing: it opens a fresh page each attempt. Retrying goto on a page that already failed keeps whatever partial state the failure left behind, and on two of my domains that state was enough to poison the second attempt.

Reading the error text is what turns a retry policy into an informed one. Chromium writes a specific string into request.failure().errorText, and each string points at a different layer.

page.on('requestfailed', (req) => {
  console.log(req.failure().errorText, req.url().slice(0, 120));
});

That distinction matters for cost. An ERR_TUNNEL_CONNECTION_FAILED on one host while every other host answers normally is a target side filter, and discarding the address there throws away a working endpoint for no reason.

Level seven: running the whole thing in a container

Local runs and container runs disagree about DNS, about shared memory and about where credentials live. My image starts from the official Playwright base, which already carries the browsers and the system libraries they need.

FROM mcr.microsoft.com/playwright:v1.47.0-jammy

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

ENV PROXY_HOST=93.184.16.51 \
    PROXY_PORT=8080 \
    NO_PROXY=localhost,127.0.0.1,collector.internal \
    NODE_OPTIONS=--max-old-space-size=1536

CMD ["node", "crawl.js"]

docker run --rm \
  --shm-size=1gb \
  --dns 1.1.1.1 \
  -e PROXY_USER -e PROXY_PASS \
  crawler:latest

Three flags in that command each fix a real failure. --shm-size=1gb removes the Chromium crashes that come from the 64 MB default on a container running several contexts. --dns pins the resolver, since a container that inherits a host resolver pointing at a corporate DNS server will resolve your targets through it whatever the proxy scheme says. Passing -e PROXY_USER with no value forwards the variable from the calling shell, so the password never lands in the image layers or in the compose file.

HTTP_PROXY and HTTPS_PROXY deserve a clear statement, because their role is narrower than people expect. Playwright reads them when it downloads browser binaries and the Node or Python runtime reads them for its own HTTP calls. The browser takes its exit from the launch or context option, so setting the environment variables alone leaves your pages going out directly. I set both, and I still pass the proxy object explicitly.

Level eight: parallel contexts and the arithmetic behind them

Concurrency in Playwright has three multipliers, and people usually count only the first. Contexts multiply by pages per context, and pages multiply by the requests each page holds in flight. That product is what your endpoint sees.

const CONTEXTS = 8;
const PAGES_PER_CONTEXT = 3;

const pool = await Promise.all(
  Array.from({ length: CONTEXTS }, (_, i) =>
    browser.newContext({
      proxy: { server: `http://${hosts[i]}:8080`, username: U, password: P },
    })
  )
);

My packages allow 1000 threads, the corporate one reaches 3000, and packages do not stack: two bound machines share the same ceiling, so each gets half. That single sentence changed how I schedule nightly runs, because the staging node quietly ate half of production's budget for a week before I read it properly. Traffic volume never enters the arithmetic, since a month of access while the project runs carries no ceiling on transfer and the only thing I balance is parallelism.

Where the exits come from matters as much as how many you open. Automatic rotation inside the pool gives every fresh context a different exit with no list management on my side, which suits page collection where sessions are short and disposable. Long lived sessions get the opposite treatment: one context, one exit, held for the duration.

Memory is the quiet constraint. Each Chromium context costs me 40 to 70 MB at rest and considerably more with a heavy page open, so a node with 4 GB free comfortably holds around 20 contexts with route filtering on, and roughly 12 without it.

Level nine: proving which exit the site actually saw

Everything above is configuration. This section is verification, and I run it at the start of every job, because a proxy setting that silently fails is worse than one that throws.

The cheapest probe is one navigation per context to an address echo before any real work begins.

async function exitOf(context) {
  const page = await context.newPage();
  const res = await page.goto('https://ifconfig.co/json', { timeout: 20000 });
  const body = await res.json();
  await page.close();
  return { ip: body.ip, server: res.serverAddr() };
}

def exit_of(context):
    page = context.new_page()
    res = page.goto("https://ifconfig.co/json", timeout=20000)
    data = res.json()
    page.close()
    return data["ip"], res.server_addr()

response.serverAddr() is the part worth knowing about. Through a tunnel it reports the address Chromium actually connected to, which is your endpoint, while the JSON body reports the address the target saw. Two matching values from a direct run and two different values from a proxied run is the signature you want. When the body returns your own machine's address, the proxy object never reached the engine, and the usual cause is a context created before the proxy was added to the options object.

For anything deeper I record a HAR file and read it after the run.

const context = await browser.newContext({
  proxy: { server: 'http://93.184.16.51:8080', username: U, password: P },
  recordHar: { path: 'run.har', content: 'omit' },
});

The HAR gives me per request timings, the exact headers the browser sent, and every failed entry with its status. I grep it for x-forwarded-for and via on the response side, since a filtering layer at the target often names the reason for a block in a header long before it starts serving captchas. Those same runs feed my journal: date, batch id, contexts, median time to first byte, share of requests that failed, and the error text that dominated. After a month the journal told me something no single run could, which is that failures clustered by hour of day and stayed flat across address batches. That pointed at the target's own rate reaction, and the fix was scheduling. My collection jobs now sit on addresses set aside for scraping work with a schedule shaped by that journal, and the failure share went from 9 percent to under 2.

One more probe belongs here. After a block, repeat the same navigation with page.request.get() from inside the same context: the API request reuses the context cookies and the context proxy while skipping the rendering layer. A 200 from the API call against a blocked page tells me the filter reacted to browser behaviour, and a matching block on both tells me it reacted to the address.

Related material on this site: the Puppeteer proxy configuration guide covers the same nine levels for the other Node driver, reading block headers explains the response fields I grep the HAR for, and proxy switching in Chrome with SwitchyOmega handles the manual side when you are reproducing a crawler failure by hand. For the arithmetic in level eight, the pool calculator works out how many concurrent threads and how many addresses a given page count and schedule will need.