Guides
Each guide covers one tool or one job from first launch to a finished run, with the errors that show up along the way and what each one points at.
Browsers and desktop
Proxifier SOCKS5 Setup
Windows has no single switch that pushes every program through a proxy. Chrome picks up the system settings, curl reads an environ
guideSwitchyOmega proxy setup
SwitchyOmega sits between Chrome and the browser proxy configuration, and it decides one thing that matters: which requests leave
guidePlaywright proxy setup
Playwright hands you three places to attach a proxy, and the place you pick decides what the rest of the run is able to do. I have
guidePuppeteer proxy authentication
I keep a small fleet of Puppeteer workers that pull catalog pages and search results into my own reports. The first version of tha
guideAdsPower proxy setup
I keep 60 to 80 AdsPower profiles alive at any given time. The browser handles the fingerprint side of that work. The address side
guideSelenium proxy authentication
I keep a set of Selenium jobs that sign into supplier portals and export order tables nobody publishes through an API. The first v
Code and libraries
Scrapy proxy middleware
Scrapy ships with proxy support already wired in. Most people meet it once, set an environment variable, watch a crawl work, and n
guidecurl through a proxy
Almost every proxy question I have chased this year was answered by one curl command typed into a terminal. The command itself is
guidePython requests proxy session retry
My first proxied script in Python was four lines long. One requests.get, one proxies dict, one print. It worked, I felt fine about
guideNode.js HTTP request through a proxy
Three HTTP clients cover most of the Node code I maintain. axios sits in the older services, got runs the collection scripts, and
guideGolang HTTP proxy transport
Go hands you a proxied request in three lines and then leaves you alone with everything that follows. My first version was exactly
guidePHP cURL proxy
My collection code in PHP started as a single curl_setopt call and one hard coded endpoint string. It fetched a page, printed the
Diagnostics
How to tell what blocked my request
A status code gives you the verdict. The header block above it names the party that issued the verdict, and those are two differen
guidemitmproxy inspect requests
The request that leaves your socket and the request your code describes are two different objects. Libraries add headers you never
guideProxy checker script
Every collection job I run starts with a text file. A few hundred lines of IP:PORT, sometimes IP:PORT:LOGIN:PASS, pulled fresh fro
guideGitHub Actions proxy setup
One of my nightly workflows pulls about 4 000 catalogue pages and drops them into a bucket. It ran green for eleven nights. On the
Collection jobs
Scrape product pages at scale
Last quarter I took a catalogue with a shade over 200 000 items. The brief read like every other one: title, price band, availabil
guideScrape job boards
I keep a daily picture of one national job aggregator for a labour analytics group. The open set sits around 412 000 postings on a
guideScrape business directories
A regional directory reached me with a plain brief. Every organisation in the city as one row, with phones, hours and an activity
guidePrice monitoring pipeline
I keep a price monitor running for an appliance and power tool retailer. Forty thousand of our own positions, nine competitor site