Selenium proxy authentication: getting the password past ChromeDriver, geckodriver and Grid

I keep a set of Selenium jobs that sign into supplier portals and export order tables nobody publishes through an API. The first version ran fine on my laptop and died on the build box inside a minute. Chrome came up, the first navigation ended on ERR_TUNNEL_CONNECTION_FAILED, and the endpoint string in the arguments was correct down to the last character.

The split is structural. Selenium hands the endpoint to the browser as a startup switch, the browser reads scheme, host and port out of that switch, and the login pair belongs to a layer the switch never touches. Firefox divides the same work along a different line and keeps its network preferences inside a profile. Grid adds a third place where a setting stops travelling without telling you.

Below is the path I walk for every new job now: the Chrome switch and what it carries, the exact reason a login in the launch string dies, the three routes that get credentials accepted, the Firefox profile and its own preferences, remote sessions on Grid, headless against a window on screen, proving which address the site actually saw, and the startup errors that look identical until you read them closely. Code samples come in Python and Java, both pulled from jobs that are running while I write this.

ChromeOptions and the switch that carries the address

Chrome takes the endpoint as a process argument. It applies to the browser as a whole, so every tab, every iframe and every background request from that instance leaves through the same hop.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument('--proxy-server=http://45.153.14.62:8000')
opts.add_argument('--proxy-bypass-list=<-loopback>')
opts.add_argument('--disable-background-networking')
opts.add_argument('--disable-blink-features=AutomationControlled')
opts.add_experimental_option('excludeSwitches', ['enable-automation'])

driver = webdriver.Chrome(options=opts)
driver.set_page_load_timeout(45)

ChromeOptions opts = new ChromeOptions();
opts.addArguments("--proxy-server=http://45.153.14.62:8000");
opts.addArguments("--proxy-bypass-list=<-loopback>");
opts.addArguments("--disable-background-networking");
opts.setExperimentalOption("excludeSwitches", List.of("enable-automation"));

WebDriver driver = new ChromeDriver(opts);
driver.manage().timeouts().pageLoadTimeout(Duration.ofSeconds(45));

Three of those lines earn their place for reasons that took me a while to see.

Chrome skips loopback and plain local names by default, so my own test server on port 3000 answered directly while the rest of the traffic went out through the hop. That made my early debugging confusing, because half of what I was watching never touched the endpoint at all. --proxy-bypass-list=<-loopback> removes the exception.

--disable-background-networking keeps Chrome from talking to its update and telemetry hosts through my endpoint. Those calls are tiny. They still open sockets. Sockets are the unit a package sells, so I count them.

Selenium also carries a W3C capability for the same job, and the difference matters once Grid enters the picture. The capability travels inside the session payload; the switch travels inside the browser command line. Both reach the browser on a local run, and only one of them survives certain Grid setups untouched.

from selenium.webdriver.common.proxy import Proxy, ProxyType

p = Proxy()
p.proxy_type = ProxyType.MANUAL
p.http_proxy = '45.153.14.62:8000'
p.ssl_proxy = '45.153.14.62:8000'
p.no_proxy = 'localhost,127.0.0.1'
opts.proxy = p

The W3C proxy capability has no field for a username or a password. Older bindings had socksUsername and socksPassword, the current specification dropped them, and anything you attach there today goes nowhere. Here is where each piece of the configuration actually lives across the stack.

That last row is the one that decides how a fleet gets built. My endpoint list arrives from the dashboard as IP:PORT and IP:PORT:LOGIN:PASS, and the second shape is useless until something in the browser is ready to answer a challenge. The pool behind those lines holds around 12000 live entries of server grade IPv4 addresses, refreshed as machines come and go, so my scheduler treats the entries as interchangeable and pulls a new one per job.

Where the credential pair disappears in Chromium

The failing form is the one everyone writes first, because curl accepts it and half the tutorials on the internet show it.

#parses fine, fails on every request
opts.add_argument('--proxy-server=http://u17402:k93qmz@45.153.14.62:8000')

#https targets:  net::ERR_TUNNEL_CONNECTION_FAILED
#plain http:     407 Proxy Authentication Required, body readable

Chromium treats the value as a proxy URI and discards the userinfo component before the network stack ever sees it. No warning appears in the console, no launch error fires, and no switch turns the behaviour off. The browser then meets a 407 with no handler registered for it and the navigation dies where it stands.

The two error shapes are worth separating, because they are the fastest signal I have. On an https target the failure surfaces as a tunnel error, since the CONNECT never completes and there is no response body to render. On a plain http target you get the 407 page itself, which you can read with driver.page_source. An endpoint that is genuinely down answers ERR_PROXY_CONNECTION_FAILED on both schemes, so the pair of symptoms tells an authentication problem apart from a dead hop in about four seconds.

SOCKS5 has a harder edge in Chrome. The browser carries no username and password negotiation for SOCKS at all, so there is nothing for a handler to answer and nowhere in the switch to put the pair. On Firefox the same endpoint accepts a login through the profile, which is one of the few places where geckodriver has the easier job.

Selenium itself offers no equivalent of the per page authentication call that other automation libraries expose. The classic API stops at the WebDriver protocol, and the protocol has no verb for a proxy challenge. Everything below is a way around that gap.

Three routes that get the password accepted

Binding the egress address so no challenge appears

The quietest route removes the challenge from the picture. You bind the machine's own outbound address in the dashboard, the endpoint recognises the connection, and no 407 is ever sent. Chrome then needs the plain IP:PORT form and nothing else, which means no extension, no local hop and no extra moving part in the container image.

A package carries 2 bound addresses and they can be swapped whenever a worker moves, so a build agent and a laptop can both be live. One piece of arithmetic surprised me here: thread capacity attaches to the package, and with both slots in use the limit splits between them. A package rated at 1000 threads gives 500 per bound address. A Selenium fleet has to be sized against that half, since every browser holds several sockets at once.

Binding is what I use for anything running inside my own network. It fails the moment a job moves to a runner whose address changes per build, and that is where the other two routes start.

An unpacked extension that answers the 407

Chrome will answer a proxy challenge if something inside the browser handles onAuthRequired. An extension does that in about thirty lines, and Selenium can load one at startup.

import os, json, zipfile, tempfile

MANIFEST = {
    "manifest_version": 2,
    "name": "proxy-auth",
    "version": "1.0",
    "permissions": ["proxy", "webRequest", "webRequestBlocking", "<all_urls>"],
    "background": {"scripts": ["bg.js"]},
}

BG = """
chrome.webRequest.onAuthRequired.addListener(
  function (details) {
    return { authCredentials: { username: "%s", password: "%s" } };
  },
  { urls: ["<all_urls>"] },
  ["blocking"]
);
"""

def auth_extension(user, password):
    path = os.path.join(tempfile.mkdtemp(), 'proxy_auth.zip')
    with zipfile.ZipFile(path, 'w') as z:
        z.writestr('manifest.json', json.dumps(MANIFEST))
        z.writestr('bg.js', BG % (user, password))
    return path

opts.add_argument('--proxy-server=http://45.153.14.62:8000')
opts.add_extension(auth_extension('u17402', 'k93qmz'))

ChromeOptions opts = new ChromeOptions();
opts.addArguments("--proxy-server=http://45.153.14.62:8000");
opts.addExtensions(new File("/opt/jobs/proxy_auth.zip"));
WebDriver driver = new ChromeDriver(opts);

Four notes from my own broken runs. The extension has to be built per endpoint if the credentials differ, so my worker writes a fresh archive into a temp directory at startup and deletes it on teardown. Manifest v2 still loads for an unpacked local extension, and a v3 port moves the listener into a service worker that the browser is free to suspend, which turns a stable job into an occasional 407 at page 30. Old headless mode refused extensions altogether, so this route and --headless were mutually exclusive for years. And the credentials sit in a file on disk in plain text, which is fine on a machine I own and unacceptable on a shared runner.

There is a fourth path worth knowing, a CDP one: Fetch.enable with handleAuthRequests set to true, then answer Fetch.authRequired with the pair. Selenium 4 exposes CDP through execute_cdp_cmd, so it works without an extension file. I keep it in reserve for headless jobs where I want no artefacts on disk, and I accept that a CDP session bound to the driver adds one more thing that can drop mid run.

selenium-wire and a local hop that holds the pair

The third route puts a small local proxy between the browser and the endpoint. Chrome dials the local port with no authentication, the local hop opens the upstream connection and presents the credentials there.

from seleniumwire import webdriver

wire = {
    'proxy': {
        'http': 'http://u17402:k93qmz@45.153.14.62:8000',
        'https': 'http://u17402:k93qmz@45.153.14.62:8000',
        'no_proxy': 'localhost,127.0.0.1',
    },
    'verify_ssl': False,
}

driver = webdriver.Chrome(seleniumwire_options=wire, options=opts)
driver.get('https://target.example/orders')

for r in driver.requests:
    if r.response and r.response.status_code != 200:
        print(r.response.status_code, r.url)

The request log in that loop is the reason I reach for this library at all. Every request the page made, with status and headers, available in Python without a CDP dance. On a portal that silently returns 403 for one XHR out of forty, that log found the failing call in a single run.

The cost is real and worth stating plainly. The library terminates TLS locally and re-signs it with its own certificate, so the browser sees a certificate chain no site issued, and any pinning or fingerprint check on the far side sees a TLS handshake produced by Python. For collection jobs on ordinary pages that has never bothered me. For anything sensitive to how the handshake looks, the extension route keeps the browser's own TLS intact, and profile tooling such as Dolphin Anty driven through a proxy handles the fingerprint side properly while Selenium attaches over the debugging port.

Here is how the three compare on the things I actually check before picking one.

Firefox profile and the network preferences it keeps

Firefox ignores command line proxy switches entirely. Everything lives in preferences, and Selenium 4 sets them straight on the options object with no separate profile directory needed.

from selenium.webdriver.firefox.options import Options as FxOptions

fx = FxOptions()
fx.set_preference('network.proxy.type', 1)          # 0 direct, 1 manual, 2 pac, 4 auto, 5 system
fx.set_preference('network.proxy.http', '45.153.14.62')
fx.set_preference('network.proxy.http_port', 8000)
fx.set_preference('network.proxy.ssl', '45.153.14.62')
fx.set_preference('network.proxy.ssl_port', 8000)
fx.set_preference('network.proxy.no_proxies_on', 'localhost, 127.0.0.1')
fx.set_preference('network.proxy.failover_direct', False)
fx.set_preference('media.peerconnection.enabled', False)
fx.set_preference('signon.autologin.proxy', True)

driver = webdriver.Firefox(options=fx)

FirefoxOptions fx = new FirefoxOptions();
fx.addPreference("network.proxy.type", 1);
fx.addPreference("network.proxy.socks", "45.153.14.62");
fx.addPreference("network.proxy.socks_port", 1080);
fx.addPreference("network.proxy.socks_version", 5);
fx.addPreference("network.proxy.socks_remote_dns", true);
fx.addPreference("network.proxy.failover_direct", false);
WebDriver driver = new FirefoxDriver(fx);

Four preferences in there do work that no Chrome switch does.

network.proxy.failover_direct defaults to true, which means a failed hop lets the browser retry the same request with no proxy at all. My own address then appears in the target's log while the job reports success. I set it to false in every profile I build, and I consider that line the single most valuable one on this page.

network.proxy.socks_remote_dns decides who resolves the hostname. With it off, the local resolver looks up every domain the job touches and the lookups leave the machine directly. With it on, the name travels inside the SOCKS request and the endpoint resolves it. For a SOCKS5 job I always turn it on, and the same choice sits per profile in an antidetect browser with its own network settings, where the endpoint handles name resolution along with the traffic.

media.peerconnection.enabled set to false shuts down WebRTC. A page that runs a peer connection can read local interface addresses through it, and no proxy setting covers that path. Chrome needs the same treatment through a policy or an extension; in Firefox it is one line.

signon.autologin.proxy stops Firefox from re-prompting once a stored login for the hop exists. It does nothing on a fresh profile with no stored login, which is the part that confuses people who copy the preference out of a forum post. geckodriver has no way to type into the native proxy dialog, so on Firefox the same three routes apply: bind the address, load an .xpi with an onAuthRequired listener, or run the local hop.

One habit that saved me hours: build the profile once, launch Firefox against it by hand, open about:config and confirm the values took. A misspelled preference name is accepted silently by geckodriver and produces a browser that goes out direct.

Grid: attaching an endpoint to a remote session

On Grid the browser runs on a node, so the traffic leaves from the node's address. This is the detail that ends most Grid debugging sessions once someone says it out loud. My binding pointed at my workstation, the node sat in a container on another host, and every request answered 407 while identical code passed locally.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument('--proxy-server=http://45.153.14.62:8000')
opts.add_argument('--no-sandbox')
opts.add_argument('--disable-dev-shm-usage')
opts.set_capability('se:name', 'orders-export')

driver = webdriver.Remote(
    command_executor='http://grid.internal:4444',
    options=opts,
)

ChromeOptions opts = new ChromeOptions();
opts.addArguments("--proxy-server=http://45.153.14.62:8000", "--no-sandbox");
WebDriver driver = new RemoteWebDriver(
    new URL("http://grid.internal:4444"), opts);

Everything inside options is serialised into the session request and applied on the node, switches included. An extension file is the exception: add_extension sends the archive as base64 inside the payload and that does travel, while --load-extension points at a path that has to exist on the node's filesystem. I learned this by watching a job pass on my machine and fail on Grid with no error message at all, the browser simply going out direct because the extension directory was absent.

Three more things I check on any Grid setup that routes through an endpoint.

The node image needs the same flags a container always needs. --no-sandbox and --disable-dev-shm-usage are there because the default shared memory allotment runs out on a heavy DOM and the renderer dies mid navigation, which surfaces as a detached frame error that says nothing about memory.

Session concurrency is thread arithmetic. A node configured for 5 sessions, times three nodes, times roughly 6 sockets per browser, puts 90 concurrent connections on the endpoint. Against 500 threads on a bound address there is room; against a node count someone doubled last week there may not be, and the endpoint simply refuses the extra connections while the platform reports nothing.

Address rotation belongs to the client, since the node knows only what it was handed. My scheduler picks the entry before it builds the options object, one job to one entry from first request to last. The pool is a worldwide mix with rotation handled inside the pool, so my side of that arrangement stays a simple cursor over the current list.

pool = [l.split(':') for l in open('pool.txt').read().split()]
cursor = 0

def next_endpoint():
    global cursor
    host, port, *cred = pool[cursor % len(pool)]
    cursor += 1
    return host, port, cred

Swapping the exit address inside a session is the failure that produces no error. The portal handed me a cookie on page one, page 27 arrives from a different address, the session is invalidated, and the response is a 200 carrying the layout a signed out visitor sees. Empty columns in the export, nothing in the log.

Headless against a window on screen

Headless is the default on build machines and the wrong default for some targets. My measurements on the same job, 40 pages, three passes, on a worker box with 8 GB of memory:

New headless mode changed my defaults. The old one ran a separate lightweight browser build with a different feature set, and the missing extension support made the credential route unavailable, which is why so many older guides pair headless with selenium-wire. The new mode runs the same browser with no visible window, so the extension loads and the handler answers.

Where a window still wins for me is on portals that check the rendering surface. A job that reported an empty product grid under headless returned full data under Xvfb with an identical configuration, and the difference showed up in nothing except the result. I keep both modes available behind one environment variable and switch a job over when the output looks thin.

import os
if os.environ.get('HEADLESS', '1') == '1':
    opts.add_argument('--headless=new')
    opts.add_argument('--window-size=1366,768')

Window size deserves a line. Headless Chrome starts at a small default viewport, and a responsive site served to that viewport returns the mobile layout with different selectors. Half the "my selectors broke in CI" reports I have looked at were this and nothing else.

Reading the exit address from inside the driver

Any check that runs outside the browser tells you about the machine. The address the site sees comes from the same session that does the work, so the check has to run through the driver.

import json

def exit_address(driver):
    driver.get('https://api.ipify.org?format=json')
    raw = driver.execute_script('return document.body.innerText')
    return json.loads(raw)['ip']

addr = exit_address(driver)
print('session exits on', addr)
assert addr != LOCAL_ADDRESS, 'traffic went out direct'

driver.get("https://api.ipify.org?format=json");
String raw = (String) ((JavascriptExecutor) driver)
    .executeScript("return document.body.innerText");
System.out.println("session exits on " + raw);

The assertion is the part I care about. A job that quietly goes direct produces perfectly good looking data and puts my own address in someone's access log, and Firefox with default failover behaviour does exactly that on a flaky hop. I run this check at the top of every job, before the first real navigation, and abort the run when it comes back wrong.

Three checks sit alongside it in my starter routine. A second call to the same endpoint after the first page confirms the address held for the session. A request to a header echo service shows what the hop appends, and an endpoint that adds Via or X-Forwarded-For announces itself to every target; the exits I keep for header sensitive work pass the request through untouched, which I verify once per pool refresh. And a DNS check page tells me which resolver answered, which is how I caught a Firefox profile where I had set the SOCKS host and forgotten socks_remote_dns.

For the timing side I read the browser's own numbers through the Navigation Timing API, since they separate the hop from the target's thinking time.

t = driver.execute_script("""
  const n = performance.getEntriesByType('navigation')[0];
  return {connect: Math.round(n.connectEnd - n.connectStart),
          ttfb: Math.round(n.responseStart - n.requestStart),
          total: Math.round(n.loadEventEnd - n.startTime)};
""")

connect covers the TCP and CONNECT work with the hop. ttfb is mostly the target thinking. When someone tells me the collector got slow, those two numbers say which side to look at, and twice now they showed a target that had added a redirect on every page.

Startup errors and what each one points at

Most Selenium proxy failures produce one of a dozen messages, and several of them look alike while pointing at different layers. This is the table I keep open while debugging.

Two of those deserve more than a row.

DevToolsActivePort file doesn't exist is the message that sends people down the longest wrong path, because it reads like a driver problem and comes from the browser refusing to start. In a container it is almost always the sandbox or shared memory. I pass --no-sandbox, --disable-dev-shm-usage and give the container a full gigabyte of shared memory, and the message has not returned since.

The version mismatch line changed shape recently. Selenium Manager now downloads a matching driver on its own, so a pinned driver binary in an image is the usual cause; deleting the pinned copy and letting the manager work fixed a build that had been failing every Monday after browser updates landed.

One habit for the whole table: log the full driver capabilities returned at session start. driver.capabilities echoes back what the browser accepted, and comparing that to what you sent catches a dropped proxy field before the first page ever loads.

The setup I keep across projects

Bound egress on machines I own, an extension handler on runners with a changing address, selenium-wire when I need the request log, failover_direct off in every Firefox profile, one endpoint per job held from the first request to the last, an exit address assertion before the first navigation, and the timing pair logged on every page. None of the pieces are clever. The gain came from knowing which layer owns which setting: the process owns the address, a handler owns the credentials, the profile owns Firefox's whole network behaviour, the node owns the address a Grid job actually leaves on. When a job needs a full browser identity around it as well, I run it against proxy setup for Dolphin profiles and attach Selenium to the profile over the debugging port, which keeps the fingerprint work and the driver work in separate hands.

If you are carrying the same job across tools, my write up on driving Playwright through a proxy covers the context level API that removes most of this ceremony, and Puppeteer and the launch flag walks the same ground on the Node side where a per page authentication call exists. For profile based work with fingerprints attached, setting up AdsPower profiles shows the fields a browser like that expects. And when an endpoint arrives in a shape you have not seen before, the connection string parser will split it into the parts each of these tools wants.