GitHub Actions proxy setup: environment variables, secrets and runners that stay reachable

One of my nightly workflows pulls about 4 000 catalogue pages and drops them into a bucket. It ran green for eleven nights. On the twelfth it went red with a single line I had never seen from that job: curl: (56) Received HTTP code 407 from proxy after CONNECT. Nothing in the repository had changed that day. What changed was the runner. A colleague had moved the job off my own machine onto ubuntu-latest to speed up the queue, and my access was tied to the source address of the machine that no longer ran the job.

That single line is the whole subject of this article. Proxy access has to know who is calling. In continuous integration the caller is a machine that may not exist yet when you write the workflow, and on cloud runners it gets a fresh address every time it boots. Everything below follows from that: which variables the tools read, where the login lives so it never lands in a log, what a container step does to your settings, why the dependency cache quietly goes around the hop, and how to read a failure at three in the morning without guessing.

The YAML below is copied from four repositories, two on hosted runners and two on machines in my own rack, with hostnames changed.

What the three variables control, and the fourth one for SOCKS5

HTTP_PROXY applies to destinations whose scheme is http://. HTTPS_PROXY applies to destinations whose scheme is https://, and the value itself is almost always a plain http:// endpoint, since the client opens a CONNECT tunnel through it. People read HTTPS_PROXY=http://... as a typo and correct it to https://, then spend an hour on handshake errors. The scheme in the value describes how you reach the hop. The name of the variable describes what you are reaching for.

NO_PROXY is a comma separated list of destinations that skip the hop entirely. Its behaviour varies by implementation more than any other piece of this puzzle. Common ground: bare hostnames match, a leading dot means "this domain and everything under it", localhost and 127.0.0.1 almost always need to be listed by hand, and a single * disables proxying everywhere. Everything else is negotiable. Go accepts CIDR blocks like 10.0.0.0/8. curl accepts suffix matching and, since a fairly recent version, CIDR too. Python's urllib does simple suffix matching with no CIDR at all. Java ignores the variable and wants http.nonProxyHosts with pipe separators.

ALL_PROXY carries the SOCKS5 case. curl, Python's httpx, and a fair number of Go clients read it; when the value starts with socks5h:// the hostname is resolved by the endpoint at the far end, and with socks5:// it is resolved on the runner before the connection opens. That distinction produces failures that look like dead endpoints, and I have chased it twice.

Case matters and the rule is not symmetric. curl deliberately ignores an uppercase HTTP_PROXY because a CGI environment can be poisoned by a client sending a Proxy: request header, which lands in the process environment under that exact name. Lowercase http_proxy works. Both cases work for https_proxy. Most other tools accept either. I set both cases everywhere and stopped thinking about it.

name: nightly-collect
on:
  schedule:
    - cron: "17 2 * * *"
  workflow_dispatch:

env:
  http_proxy: http://gw.example.net:3128
  HTTP_PROXY: http://gw.example.net:3128
  https_proxy: http://gw.example.net:3128
  HTTPS_PROXY: http://gw.example.net:3128
  no_proxy: localhost,127.0.0.1,::1,.internal.example.net,169.254.169.254
  NO_PROXY: localhost,127.0.0.1,::1,.internal.example.net,169.254.169.254

The 169.254.169.254 entry earns its place. Cloud runners talk to a metadata endpoint there for their own housekeeping, and sending those calls through a hop produces timeouts far away from anything your job does.

Which tools read the variables and which ignore them

I keep this table in the repository wiki because every new dependency raises the same question. The middle column is the one that decides how much YAML you write.

Two rows in that table cost me the most time. The Node one is the reason a script that worked under axios stopped working after a rewrite to the built in fetch, with no error mentioning a proxy at all, only a connect timeout to the origin. The Docker one is the reason docker build inside a job can hang while every other step in the same job runs fine.

Where the login lives, and how it stays out of the run log

Our access comes in two forms. Either the source address of the caller is registered in the panel, or the caller presents a login and password. The list downloads as IP:PORT for the first form and IP:PORT:LOGIN:PASS for the second. On cloud runners only the second form is available, which puts credentials into the workflow, which puts them one careless line away from a public log.

Repository secrets are masked in the log by the runner, but only the exact string. A password of Sx7-fVq2 gets replaced with three asterisks. The same password inside a URL, url encoded, with the hyphen intact and the rest of the URL around it, is a different string, and the masker has never seen it. That is the leak. It happens the moment you build http://user:pass@host:port inside a step and echo it, or run a command under set -x, or pass it to a tool that prints its own configuration on startup.

The fix is three lines. Compose the URL in a step, register it with the masker yourself, then hand it to the following steps through the environment file.

      - name: Compose the proxy URL and mask it
        env:
          PX_HOST: ${{ secrets.PROXY_HOST }}
          PX_PORT: ${{ secrets.PROXY_PORT }}
          PX_USER: ${{ secrets.PROXY_USER }}
          PX_PASS: ${{ secrets.PROXY_PASS }}
        run: |
          enc_user=$(python3 -c 'import urllib.parse,os;print(urllib.parse.quote(os.environ["PX_USER"],safe=""))')
          enc_pass=$(python3 -c 'import urllib.parse,os;print(urllib.parse.quote(os.environ["PX_PASS"],safe=""))')
          url="http://${enc_user}:${enc_pass}@${PX_HOST}:${PX_PORT}"
          echo "::add-mask::${url}"
          echo "::add-mask::${enc_pass}"
          {
            echo "http_proxy=${url}"
            echo "HTTP_PROXY=${url}"
            echo "https_proxy=${url}"
            echo "HTTPS_PROXY=${url}"
          } >> "$GITHUB_ENV"

Four details in that block earned their place. The url encoding matters because a password containing @, / or # breaks the URL parser and surfaces as "could not resolve host" with half your password sitting in the hostname position, printed in the log. The masker hears about both the full URL and the encoded password, since the encoded form differs from the secret the runner already knows. Writing to the environment file keeps the value off the command line, which matters on self-hosted machines where any local process can read /proc. And the secrets arrive through env: on the step, so they never appear inside the run script text that the runner prints.

Two more habits. Debug logging keeps masking on, so turning it on to chase something is safe. Workflows triggered by pull requests from forks receive no secrets at all, so I gate the collection job on the event type and let it skip. And a package that offers an HTTPS package with login credentials keeps this whole section to one step, since the same login works from any runner the job lands on without touching the panel.

Cloud runners hand you a different address on every run

This is the part that decides the shape of everything else, so I will be precise about it.

A GitHub hosted runner is a fresh virtual machine created for the job and destroyed after it. Its egress address comes from a large cloud range and differs between runs, sometimes between jobs of the same run, and there is no supported way to pin it. The published address ranges cover tens of thousands of entries, so "allow the whole range" is the same as "allow the internet".

Our package includes two bound addresses, changeable at any time from the panel. Two is generous for a workstation and a server. Two is nothing against a pool of ephemeral virtual machines. Even a rewrite that read the runner's current address and called the panel API before every job would fight two other facts: the bound pair is a shared account setting that other jobs depend on, and the thread ceiling splits in half the moment a second address is bound, dropping 1 000 concurrent connections to 500 per address. A job that rebinds on the fly would be changing the capacity of every other job in flight.

So the rule is simple. Hosted runner means the login form, always.

The last row catches people who assume "self-hosted" and "stable address" mean the same thing. A runner scaled up on demand from a cloud image behaves exactly like a hosted one for our purposes, unless every node in the pool leaves through one gateway you control.

A 407 in a hosted job therefore means one of three things and never a dead endpoint: the credentials are missing from the environment of that specific step, the password lost a character to url encoding, or the workflow is silently running with the address binding assumption that a self-hosted job left behind in a shared reusable workflow.

A self-hosted runner keeps its address, so binding works

On my own hardware the picture inverts, and it is the more comfortable one. The machine sits behind a gateway whose address does not change. I register that address in the panel once, and the workflow carries no credentials at all. There is no secret to mask, no url encoding, no fork restriction, nothing to rotate when somebody leaves the team. The endpoint list downloads as plain IP:PORT.

The second binding slot goes to my workstation, so the same collector runs locally during development and in CI without a configuration switch. That is the whole reason the pair exists. With both slots occupied the concurrency ceiling reads 500 per address, and I size the job accordingly: 40 workers in the collector, 8 concurrent requests each, which leaves comfortable headroom for the health pass that runs beside it.

Where the environment variables live on a self-hosted runner is worth knowing, because the workflow is no longer the only place they can come from. The runner reads a .env file in its installation directory at service start, and anything there applies to every job the machine picks up. That is convenient and it is also a trap: a variable set there overrides nothing in the workflow, yet it does apply to steps that never mention a proxy, including the runner's own calls to fetch actions. I keep that file empty and declare everything in the workflow, so that reading the YAML tells the whole story.

jobs:
  collect:
    runs-on: [self-hosted, linux, collector]
    env:
      http_proxy: http://gw.example.net:3128
      https_proxy: http://gw.example.net:3128
      no_proxy: localhost,127.0.0.1,.internal.example.net,169.254.169.254
    steps:
      - uses: actions/checkout@v4
      - name: Confirm the exit address before doing any work
        run: |
          seen=$(curl -sS --max-time 8 https://echo.internal.example.net/ip | jq -r .ip)
          echo "exit address: $seen"
          test -n "$seen"

That check costs under a second and has saved several nights. If the gateway changed its address, the job stops there with a readable message, and no worker spends twenty minutes collecting 407 responses. Running the collector from the private pool my runners dial on a machine whose address I control means the check almost always passes, and when it fails the cause is on my side and takes one panel edit. For a build pipeline that runs every night the sensible unit of access is monthly pool access for build agents, since the binding stays valid for the whole period and the workflow needs no attention between edits.

Step, job, workflow, container: where the setting belongs

There are three levels of env and they merge, with the narrowest winning. Workflow level applies to every job. Job level applies to every step of that job. Step level applies to that step alone. A value written to the environment file during a step applies to all following steps of the same job, and to nothing in any other job, since each job gets its own machine.

My rule after a few messy workflows: put the routing at job level and use step level only to take a step out of the routing. That reads well in review, because the exception is visible on the line it applies to.

jobs:
  build-and-collect:
    runs-on: ubuntu-latest
    env:
      https_proxy: ${{ env.PX_URL }}
      http_proxy: ${{ env.PX_URL }}
      no_proxy: localhost,127.0.0.1,registry.npmjs.org,pypi.org,files.pythonhosted.org
    defaults:
      run:
        shell: bash
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies, going direct to the registry
        env:
          http_proxy: ""
          https_proxy: ""
        run: npm ci

      - name: Collect
        run: node ./collect.js --workers 12

Setting a variable to an empty string is the portable way to remove it for one step. Some clients treat an empty value as "no proxy configured" and some treat it as a malformed URL, so I test that behaviour once per tool.

Container steps are where careful setups come apart. A job with a container: key runs its steps inside that image, and the variables you declared at job level do reach it, because the runner passes them to docker run. Service containers are a different matter: they are separate containers on the same network, addressed by their service name, and that name must appear in NO_PROXY or every call to your test database goes out to the hop and comes back as a resolution failure. A step that itself calls docker run passes nothing automatically, and docker build needs the values as build arguments, since the build runs in the daemon's context and not in the step's.

    container:
      image: python:3.12-slim
      env:
        https_proxy: ${{ env.PX_URL }}
        no_proxy: localhost,127.0.0.1,postgres,redis,.internal.example.net
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: ci
    steps:
      - name: Build an image that can reach the outside during build
        run: |
          docker build \
            --build-arg https_proxy="$https_proxy" \
            --build-arg no_proxy="$no_proxy" \
            --network host \
            -t collector:ci .

The Docker daemon on a self-hosted runner needs its own configuration for pulls, in a systemd drop in file. Nothing in the workflow reaches it. I found that out when a job pulled its base image fine on one machine and hung for six minutes on another, with identical YAML on both.

Dependency caches and the calls that go around the hop

Three kinds of traffic leave a CI job, and they deserve different routing. Traffic to the target you are collecting from should go through the hop, because that is the entire reason the hop exists. Traffic to package registries and to the cache backend should go direct, since routing gigabytes of wheels and tarballs through a collection endpoint adds latency to every build and provides nothing.

actions/cache talks to a storage backend over HTTPS. Whether it honours the environment variables depends on which internal client path the current version takes, and I have seen both behaviours across upgrades. The visible symptom of it going through the hop is a restore step that used to take 9 seconds and now takes 70, with a Warning: Failed to restore cache on a bad night. Guessing is pointless here. I pin the backend hostnames into NO_PROXY and measure the restore step's duration on every run.

Package managers add their own layer, because their config files win over the environment. npm config get proxy can return a value set months ago in a Docker image, and it overrides the empty string you just exported. The same is true of pip.conf, .curlrc and .wgetrc. When a step behaves as though your settings do not exist, the config file is the first place to look.

# what actually goes direct, written once and reviewed when a dependency changes
export no_proxy="localhost,127.0.0.1,::1,\
registry.npmjs.org,registry.yarnpkg.com,\
pypi.org,files.pythonhosted.org,\
proxy.golang.org,sum.golang.org,\
ghcr.io,pkg-containers.githubusercontent.com,\
objects.githubusercontent.com,\
169.254.169.254,.internal.example.net"
export NO_PROXY="$no_proxy"

# and prove the config files are not fighting the environment
npm config get proxy; npm config get https-proxy
pip config list | grep -i proxy || true
git config --get http.proxy || true

Collection traffic is the part that grows without warning. A job that pulled 4 000 pages in the first week pulls 30 000 in the second, because somebody widened a filter in a config file. Access tiers where traffic is left unmetered take that variable out of the planning entirely, and the only ceiling left is the thread count, which is a number I can reason about from the workflow's own concurrency settings. When the latency of the hop shows up in build times, endpoints living on rack hardware keep the distribution narrow enough that a slow step means something real happened.

Scheduled runs and the failures that pile up unseen

A schedule trigger is the most reliable way to accumulate silent breakage. Nobody reads a green check at two in the morning, and nobody reads a red one either until the report that depends on it comes up short.

Scheduled workflows queue on a shared pool, so the actual start time drifts by minutes and occasionally by much longer. That drift is fine for collection work and hostile to anything that assumes an exact window. GitHub also disables scheduled triggers in repositories with no commit activity for a long stretch, which is a fine policy and a nasty surprise when the collector lives in a repository that rarely changes. I push a trivial commit to the state file on every successful run, which keeps the repository active and gives me a dated record at the same time.

The thing that made night failures readable was writing one metrics line per run to the step summary. The first error gets counted and the run continues; the threshold decides the verdict.

      - name: Collect with a failure budget
        id: run
        run: |
          node ./collect.js --workers 12 --out out.jsonl --stats stats.json || true
          ok=$(jq -r .ok stats.json); total=$(jq -r .total stats.json)
          p95=$(jq -r .ttfb_p95 stats.json); c407=$(jq -r '.codes["407"] // 0' stats.json)
          rate=$(awk "BEGIN{printf \"%.3f\", $ok/$total}")
          {
            echo "| metric | value |"
            echo "|---|---|"
            echo "| pages ok | $ok / $total |"
            echo "| success rate | $rate |"
            echo "| ttfb p95 ms | $p95 |"
            echo "| 407 responses | $c407 |"
          } >> "$GITHUB_STEP_SUMMARY"
          echo "rate=$rate" >> "$GITHUB_OUTPUT"
          awk "BEGIN{exit !($rate < 0.90)}" && { echo "success rate below budget"; exit 1; }

The 407 counter has its own row for a reason. A run at 0.94 success with zero 407 responses is a normal night with a few slow origins. A run at 0.94 with 200 of them is an access problem that will be total by tomorrow, and the two look identical in a plain pass or fail. Three consecutive runs under budget open an issue automatically; one does nothing beyond the red mark.

Overlap is the other scheduled hazard. A run that hangs past its window meets the next one, both open connections against the same thread ceiling, and the second half of both fails on capacity. A concurrency group with cancellation fixes that in two lines.

Reading the run log when the job fails

Most CI failures around a hop produce one of about a dozen lines. Learning to map them saves the twenty minutes that otherwise go into re-running with debug logging on.

The ordering of that table follows the ownership question, which is the only question worth asking first: is this my account, the path, or the site I am collecting from. A 407 and a 429 both turn a run red and share nothing else. The 407 is answered in the panel in a minute. The 429 is answered by changing the job's pacing, and a re-run without that change repeats the result exactly.

Two notes on getting more out of a log. Debug logging adds the toolkit's own HTTP diagnostics, which show whether an action picked up your proxy settings at all. And curl -sS -w with a format string turns any step into a timing probe, printing connect, appconnect and starttransfer separately, so a slow night gets attributed to the hop or to the origin without argument.

Re-runs, idempotency, and what a second attempt must not repeat

The re-run button is the most used feature in any CI setup that touches a network, and it is safe only if the job was written to be re-run. Mine were not, at first. The first re-run of a partially completed collection produced a duplicate set of rows in the output bucket, and the report built on top of it counted several thousand items twice.

Three properties make a re-run harmless, and none of them cost much.

The output path carries the run identity. I name artifacts with the run id and the attempt number, so a second attempt writes beside the first and never over it. Comparing the two afterwards is often the fastest way to understand what went wrong, since the difference between attempt one and attempt two is exactly the set of pages that depend on timing.

The work list is derived, never carried. At the start of every attempt the job reads what already exists in the bucket, subtracts it from the full target list, and works on the remainder. A re-run of a job that was 80 percent done finishes the last fifth in two minutes. This also makes the failure budget honest, because the second attempt is measured against the work it actually attempted.

Retries live at two levels with different jobs to do. Inside the collector, a per request retry handles transient tunnel drops: two attempts, exponential spacing, jitter, and a hard rule that a 407 never retries. Retrying a 407 is pure waste, since nothing about the access will differ 400 ms later, and a few thousand of those in a tight loop is how a job reaches its time limit with an empty output file. At the workflow level, a rerun of failed jobs handles everything else, and the derived work list keeps it cheap.

The last piece is state that survives the runner. Hosted machines are destroyed after the job, so anything you want tomorrow goes somewhere durable: the bucket, an artifact, a committed state file. I write one line per run with the date, the pages fetched, the success rate, the p95 first byte time and the count of each status code. Six weeks of those lines answer questions no single run can, such as whether tonight's slowness is the origin reacting to my pacing or the ordinary drift I have been ignoring. That record is also what tells me when the access period is worth extending: a pipeline holding a steady rate over a month of nights has earned a month of pool access as a standing line in the setup, with no per run thinking at all.

If you want the neighbouring pieces of this setup, the pass I run beside the collector is described in building a checker for your endpoint list, and it is what feeds the working list this workflow consumes. When a job fails in a way the table above does not cover, recording proxied traffic with mitmproxy puts the actual bytes in front of you, which settles arguments about who returned what. The status codes and header sets that separate an access fault from an origin decision are laid out in what the headers behind a block tell you. And before you paste anything into a secret, the connection string parser shows how your IP:PORT:LOGIN:PASS line maps onto a URL with the encoding applied, which removes the single most common cause of a 407 in a workflow that looks correct.