Puppeteer proxy authentication: from the launch flag to a run that finishes

I keep a small fleet of Puppeteer workers that pull catalog pages and search results into my own reports. The first version of that fleet held up for about forty minutes. Then every page came back blank and the log filled with one repeated line about a tunnel that could not be established. The proxy string in the launch arguments was right, character for character, and the same credentials had worked in curl on the same machine two minutes earlier.

The cause was structural. Puppeteer hands the proxy to Chromium as a process argument, and Chromium reads that argument narrowly: scheme, host, port. Anything else in the string is dropped without a line in the console. The login and the password belong to a different layer, on the page object, and they have to be in place before the first navigation call.

What follows is the whole path I walk for every new worker now. The launch flag and its limits, where credentials actually go, how many browsers to run against how many addresses, changing the exit address between jobs, interception to cut weight, response handling with a second attempt, containers and functions, and the timing journal I use to find the slow phase. Every code sample is JavaScript pulled from workers that are running while I write this.

The --proxy-server flag and what it actually carries

The flag goes into args on launch. It applies to the browser process as a whole, so every page, every iframe and every background request from that Chromium instance leaves through the same endpoint.

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
  headless: true,
  args: [
    '--proxy-server=http://45.153.14.62:8000',
    '--proxy-bypass-list=<-loopback>',
    '--disable-background-networking',
    '--disable-features=IsolateOrigins,site-per-process',
  ],
});

Three details in that block cost me time before I understood them.

Chromium bypasses loopback and local names by default. On my laptop that meant my own test server on port 3000 answered directly while everything else went through the endpoint, which made the early debugging confusing. --proxy-bypass-list=<-loopback> removes that exception and pushes loopback through the proxy as well, which is what I want when I am checking whether the endpoint is alive at all.

--disable-background-networking stops Chromium from talking to its own update and telemetry hosts through my endpoint. Those requests are small, they still open sockets, and sockets are the thing I pay attention to when a package gives me a fixed thread count.

The scheme in front of the host decides the transport. Here is how Chromium reads the shapes I have tried.

Name lookups deserve a line of their own. With an HTTP proxy, Chromium sends the hostname inside CONNECT and the proxy resolves it, so my local resolver sees nothing. With SOCKS5 the picture depends on the build, and I stopped guessing: I add --host-resolver-rules and force everything except the endpoint itself into a dead end, so a lookup that escapes the tunnel fails loudly during a test run. Silent leakage during a real job is the outcome I am buying protection from.

args: [
  '--proxy-server=socks5://45.153.14.62:1080',
  '--host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE 45.153.14.62',
]

My endpoint list comes out of the dashboard in two shapes, IP:PORT and IP:PORT:LOGIN:PASS, and both drop straight into the patterns above once you know which half of the string Chromium keeps. The pool behind those lines holds around 12000 live entries on private datacenter IPv4 addresses, refreshed as machines come in and out, which matters later when I talk about picking a fresh endpoint per job.

Why credentials in the launch string never reach the proxy

The failing form is the one everybody writes first, because it is the form curl accepts and the form every README shows for other tools.

// looks right, fails on every request
args: ['--proxy-server=http://u17402:k93qmz@45.153.14.62:8000']

// https targets end the navigation with:
//   net::ERR_TUNNEL_CONNECTION_FAILED
// plain http targets return a body you can actually read:
//   407 Proxy Authentication Required

Chromium parses the value as a proxy URI and discards the userinfo component. There is no console warning, no launch error, no flag that turns the behaviour off. The browser then meets the 407 challenge with no handler registered for it, and the navigation dies. On https the failure surfaces as a tunnel error, because the CONNECT never completes and Chromium has no response body to show you. That difference between the two error shapes is the fastest way I know to tell an authentication problem from a dead endpoint: an endpoint that is truly down gives net::ERR_PROXY_CONNECTION_FAILED on both schemes.

SOCKS5 has a harder edge. Chromium carries no username and password negotiation for SOCKS at all, so page.authenticate has nothing to answer and the launch string has nowhere to put the pair. Two routes work here. The first is to bind the machine's own egress address in the dashboard, after which the endpoint accepts the connection with no challenge at all; a package carries 2 bound addresses and you can swap them whenever the worker moves. The second is to run the credential pair over HTTP or HTTPS and keep SOCKS5 for the workers that sit on bound machines. I use both in the same fleet: the container hosts run on binding, and the laptops I develop on use an HTTP endpoint with the login pair, because my home address changes and rebinding every morning is noise I do not need.

One arithmetic note on binding, since it surprised me. Thread capacity attaches to the package, and with 2 bound addresses the limit splits between them. A package that gives 1000 threads gives 500 per bound address once both are in use. A Puppeteer fleet has to be sized against that half.

page.authenticate: where the login and password belong

The credentials live on the page. Puppeteer registers a handler for the proxy challenge and answers it on your behalf, which is why the call has to happen before anything navigates.

const page = await browser.newPage();
await page.authenticate({ username: 'u17402', password: 'k93qmz' });
await page.goto('https://target.example/catalog?p=3', {
  waitUntil: 'domcontentloaded',
  timeout: 45000,
});

Four rules came out of my own broken runs.

Call it before goto. If the navigation starts first, the challenge arrives with no handler and the page resolves to an error before your credentials are attached. I have seen this pass locally and fail under load, because under load the ordering of two un-awaited promises stops being lucky.

It is per page. A browser with 8 tabs needs 8 calls. Nothing on the browser object sets a default.

Popups count as pages. Any target opened by the site itself starts with no credentials, and on sites that open a checkout or a login window that shows up as one mysterious blank tab in an otherwise working run.

browser.on('targetcreated', async (target) => {
  if (target.type() !== 'page') return;
  const p = await target.page();
  if (p) await p.authenticate(cred);
});

Passing null clears the credentials for that page, which I use in exactly one place: a smoke test that checks the endpoint is actually challenging connections and has not been left open by mistake.

Because I need the same four calls on every page, the worker never calls newPage directly. It calls a factory.

const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
  '(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36';

async function openPage(browser, cred) {
  const page = await browser.newPage();
  await page.authenticate(cred);
  await page.setUserAgent(UA);
  await page.setViewport({ width: 1366, height: 768 });
  page.setDefaultNavigationTimeout(45000);
  page.setDefaultTimeout(15000);
  return page;
}

The factory also solved a bug I chased for two evenings. One code path in my scheduler opened a page for a health check and skipped authentication, so a single request per cycle went out unauthenticated and the endpoint counted it as a failed attempt. Nothing broke visibly. The failure counter on that endpoint just climbed all day.

One browser per address against one browser for everything

For a long time my fleet ran the obvious layout: one Chromium process per endpoint, because the launch flag is a process level setting. With 12 endpoints that is 12 browsers, and on the box I had at the time it ate memory faster than it collected pages.

Puppeteer gives a second layout. A browser context can carry its own proxy, set at creation time, and contexts live inside one process with a shared renderer pool.

const ctx = await browser.createBrowserContext({
  proxyServer: 'http://45.153.14.62:8000',
  proxyBypassList: '<-loopback>',
});

const page = await ctx.newPage();
await page.authenticate({ username: 'u17402', password: 'k93qmz' });
await page.goto('https://target.example/catalog');
// ...work...
await ctx.close();          // sockets, cookies and cache for that endpoint go with it

I still pass a default --proxy-server at launch even when every job runs inside a context. Any page that gets opened outside a context, by a stray health check or by Chromium itself, still leaves through an endpoint and keeps the machine's own address out of the picture. That default is the cheapest guard I have against a leak I would otherwise notice weeks later in someone else's access log.

Here is what the two layouts measured like on my worker box, 8 endpoints, same job list, three passes each.

The third row is what I run. Two processes give me a blast radius I can live with, the memory stays reasonable, and switching endpoints costs a fraction of what a relaunch costs. Cookies and cache are per context, so two jobs against the same site never see each other's session, which was the property I actually needed from separate browsers in the first place.

One caveat on contexts: page.authenticate still applies per page inside the context. The proxy address travels with the context, the credentials do not.

Changing the exit address between jobs

The pool rotates on its own, so I never manage individual addresses by hand. My side of the arrangement is simpler than it sounds: pull the current list, walk it, and let the pool cycling addresses on its own handle what sits behind each entry. Since the pool is a worldwide mix with no per country selection, my scheduler treats entries as interchangeable, which makes the round robin trivial.

const fs = require('fs');

const pool = fs.readFileSync('pool.txt', 'utf8')
  .trim().split(/\r?\n/)
  .map((line) => {
    const [host, port, user, pass] = line.split(':');
    return {
      server: 'http://' + host + ':' + port,
      cred: user ? { username: user, password: pass } : null,
    };
  });

let cursor = 0;
const nextEndpoint = () => pool[cursor++ % pool.length];

The rule that keeps results consistent: one job, one endpoint, from the first request to the last. A job here means a whole logical unit, a category walk of 40 pages or a keyword batch, including any cookie the site sets on the way in. Swapping the exit address in the middle of a session invalidates whatever the site handed me on page one, and the symptom stays silent in the error log. Page 27 comes back with the layout a first time visitor sees, different sorting, shorter list, status 200 all the way through.

async function runJob(browser, job) {
  const ep = nextEndpoint();
  const ctx = await browser.createBrowserContext({ proxyServer: ep.server });
  try {
    const page = await ctx.newPage();
    if (ep.cred) await page.authenticate(ep.cred);
    const out = [];
    for (const url of job.urls) {
      out.push(await collect(page, url));
      await sleep(700 + Math.random() * 900);
    }
    return out;
  } finally {
    await ctx.close();
  }
}

Closing the context matters more than it looks. Chromium keeps proxy connections alive for reuse, and while the context is open the next job can land on a socket that was opened for the previous one. Closing it drops the pooled sockets along with the cookie jar, so the following job starts from nothing.

The pacing line at the end of the loop is deliberate too. Fixed intervals between requests produce a signature any counter on the far side can read in a few minutes. I keep a floor of 700 ms with roughly a second of jitter on top, which on a 40 page walk adds about 45 seconds and has removed more rate limiting from my logs than any header tweak I ever tried.

Request interception: dropping images and fonts to buy speed

A catalog page I collect weekly asks for 96 resources and moves 3.1 MB. I need the HTML and about 40 KB of JSON out of that. Everything else is decoration I pay for in sockets, time and bandwidth.

The direct route is interception.

const DROP = new Set(['image', 'media', 'font', 'stylesheet']);
const DROP_HOSTS = /googletagmanager|google-analytics|doubleclick|hotjar|criteo/;

await page.setRequestInterception(true);
page.on('request', (req) => {
  if (DROP.has(req.resourceType())) return req.abort();
  if (DROP_HOSTS.test(req.url())) return req.abort();
  req.continue();
});

Two warnings from my own logs. Interception switches off the HTTP cache, so a repeated run re-downloads the JS bundle every time; on a job that revisits the same host 40 times that gave back some of the speed I had just won. And a request that hits neither abort nor continue hangs until the navigation timeout fires, which is what a stray early return in that handler will do to you.

When I want the weight gone and the cache kept, I go one level down through CDP.

const cdp = await page.createCDPSession();
await cdp.send('Network.enable');
await cdp.send('Network.setBlockedURLs', {
  urls: [
    '*.png', '*.jpg', '*.jpeg', '*.webp', '*.gif', '*.svg',
    '*.woff', '*.woff2', '*.ttf',
    '*://*.googletagmanager.com/*', '*://*.doubleclick.net/*',
  ],
});

Same page, after either version: 34 requests, 640 KB transferred, DOMContentLoaded down from 4.2 s to 1.6 s. The number I care about most is the one nobody prints: concurrent sockets per page fell from about 14 to about 5.

That last figure is the whole reason I bother. Thread capacity is what a package sells, and a Puppeteer page is a socket consumer, so the arithmetic is direct. Workers multiplied by pages per worker multiplied by sockets per page has to sit under the limit. Against 500 threads on one bound address, 14 sockets per page allows 35 pages in flight; at 5 sockets it allows 100. Blocking four resource types nearly tripled the size of the fleet I could run on the same package, and the corporate tier at 3000 threads moves that ceiling to 600 pages. If most of your jobs look like mine, counting sockets the way A-Parser counts threads against a pool plus aggressive blocking will get you further than adding hardware.

Reading the response code and deciding on a second attempt

page.goto gives back a response object when the navigation reaches a server, and throws when Chromium never got that far. Those are two different failures and they need two different reactions, so my collector separates them at the top.

async function collect(page, url) {
  let res;
  try {
    res = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 45000 });
  } catch (err) {
    return { ok: false, kind: 'transport', detail: String(err.message).split('\n')[0] };
  }
  const status = res.status();
  const body = await page.content();
  if (status === 200 && body.length < 4096) {
    return { ok: false, kind: 'thin', status, bytes: body.length };
  }
  if (status !== 200) {
    return { ok: false, kind: 'http', status, retryAfter: res.headers()['retry-after'] };
  }
  return { ok: true, html: body };
}

The thin branch is the one I added last and use most. A 200 with a 3 KB body is an interstitial, a challenge page or an empty result set rendered by the site, and treating it as a success poisons the dataset without a single line in the error log. I found it by exporting a run to CSV and noticing 61 rows where the price column sat empty while the status column said 200.

Once the failure has a shape, the reaction is mechanical.

Telling the proxy hop from the origin is the part people get wrong, and the header block holds the answer. A 502 generated by a proxy arrives thin, with no Server line, no Set-Cookie, no cache headers from the target. A 502 from the origin edge carries the target's usual furniture. I log the full header set on every non-200 for that reason; it turns an argument about whose fault it is into a two second look at a log line.

My retry policy is deliberately short. One retry on a new endpoint, then the URL goes into a deferred list that runs after the main queue. Long retry chains against a blocked URL produce exactly the pattern the counter on the far side is watching for, and they stretch a 40 minute job into a two hour one for the sake of a handful of pages.

Puppeteer in Docker and in a serverless function

Containers changed two things for me, and only one of them was obvious.

The obvious one is the flag set. Chromium needs --no-sandbox under most container users, and it needs help with shared memory, because the default 64 MB of /dev/shm runs out on a page with a heavy DOM. The failure mode is a renderer that dies mid navigation and a Puppeteer error about a detached frame.

const browser = await puppeteer.launch({
  headless: true,
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',
    '--disable-gpu',
    '--proxy-server=http://45.153.14.62:8000',
  ],
});

I pass --disable-dev-shm-usage and also give the container --shm-size=1gb, because the flag pushes Chromium onto the temp directory and a busy fleet then writes a lot there. Zombie processes are the other container detail: Chromium spawns children, PID 1 in a slim image reaps nothing, and after a few hundred jobs the process table fills. An init process at the entrypoint takes care of it.

The subtle change was the address. A container's traffic leaves through the host's public address, or through a NAT gateway, or through whatever the orchestrator decides, and none of those match the address you see on your workstation. My first containerised worker answered 407 on every request while identical code passed on my laptop, because the binding in the dashboard pointed at my office address. Binding is quick to change, so the fix took a minute once I stopped reading the Puppeteer error and started reading the proxy's answer. For anything crossing a network I do not control, I keep the hop encrypted and run those workers through an HTTPS proxy endpoint with the credential pair, which drops the binding question out of the deployment entirely.

Functions add their own arithmetic. A Chromium build for a function environment ships in a package like @sparticuz/chromium, the binary unpacks into the writable temp directory on first use, and that cold start alone runs 2.5 to 4 seconds on my measurements. Keeping the browser alive between invocations is what makes the model viable.

let browserPromise = null;

async function getBrowser() {
  if (!browserPromise) {
    browserPromise = chromium.puppeteer.launch({
      args: [...chromium.args, '--proxy-server=http://45.153.14.62:8000'],
      executablePath: await chromium.executablePath(),
      headless: true,
    });
  }
  return browserPromise;
}

exports.handler = async (event) => {
  const browser = await getBrowser();
  const page = await openPage(browser, CRED);
  try {
    return await collect(page, event.url);
  } finally {
    await page.close();
  }
};

The second piece of function arithmetic is concurrency. A platform that runs 60 invocations at once runs 60 browsers, and if each one holds 5 sockets that is 300 threads against a limit of 500 on a bound address. I hit that ceiling once and spent an hour blaming my code, because the platform reported no errors and the endpoint simply refused the extra connections. The concurrency cap on that function now comes straight from the thread arithmetic, and it sits at a number I can defend.

Timing by phase and finding the slow part of the run

For a long time my only number was total job duration, which tells you a job got slower and nothing else. Phase timings changed how I tune, and the instrument is twelve lines.

function stopwatch() {
  const t0 = process.hrtime.bigint();
  let last = t0;
  const rows = [];
  return {
    mark(name) {
      const now = process.hrtime.bigint();
      rows.push({ name, ms: Math.round(Number(now - last) / 1e6) });
      last = now;
    },
    rows: () => rows,
    total: () => Math.round(Number(process.hrtime.bigint() - t0) / 1e6),
  };
}

const sw = stopwatch();
const browser = await getBrowser();
sw.mark('browser');
const ctx = await browser.createBrowserContext({ proxyServer: ep.server });
sw.mark('context');
const page = await openPage(ctx, ep.cred);
sw.mark('page');
const res = await page.goto(url, { waitUntil: 'domcontentloaded' });
sw.mark('navigation');
const data = await extract(page);
sw.mark('extract');
await ctx.close();
sw.mark('teardown');
console.log(JSON.stringify({ url, total: sw.total(), phases: sw.rows() }));

For the network side I take the browser's own view through CDP, which separates the proxy hop from the origin's thinking time. Network.responseReceived carries a timing block with proxyStart, proxyEnd, sendStart and receiveHeadersEnd. The gap between sendStart and receiveHeadersEnd is time the origin spent, with no proxy involvement in it at all. That one distinction ended a long stretch of me blaming endpoints for a target that had quietly become slow.

Here is a real median profile from one job type, 40 pages, before and after the blocking work described earlier.

The navigation row carries almost everything, which is why blocking images and fonts moved the total and why tuning the extractor never did. Inside that row my CDP numbers put the proxy hop at 60 to 140 ms, with the origin holding the rest. A worker showing 900 ms of proxy hop is a worker sitting on a distant machine in the mix, and the answer there is to close the context and take the next entry.

One habit worth copying: write the phase JSON to a file on every job and keep a week of it. When someone tells me the collector "got slow", I have a distribution to look at, and twice now that distribution showed the target had added a redirect costing 600 ms on every page.

The layout I run today

Two Chromium processes per worker box, contexts inside them, one endpoint per job, credentials set in the page factory, images and fonts blocked at the CDP level, a single retry on a fresh endpoint, and phase timings on every job. The pieces are ordinary. The difference came from understanding which layer owns which setting: the process owns the address, the page owns the credentials, the context owns the session. With those three lines straight, most of the mysterious failures stop being mysterious. In a new project I set up the endpoint list first, because a stable list of server side addresses on private hardware removes a whole class of question from the debugging, and a short trial window is enough to watch a fleet behave under real jobs.

If you are moving the same fleet across tools, my write up on driving Playwright through a proxy covers where its context level API differs from what Puppeteer gives you, and reading the headers behind a block walks through the header sets that separate a proxy hop from an origin decision. For desktop software that holds no proxy setting of its own, routing Windows applications through a proxy does the same job at the operating system layer. And when a run comes back with a code you have not seen before, the response code reference with attribution lays out who emits what and what a worker should do next.