Proxy Setup Guide

Why Proxies Matter

Datacenter IPs = INSTANT BLOCK on protected sites like Airbnb, Yelp, and anything with Cloudflare/Datadome.

Anti-bot systems check IP reputation scores. AWS, DigitalOcean, Vultr, Linode IPs have high "fraud scores" and are blocked immediately.

Proxy Types (Best to Worst for Anti-Bot)

Type	Detection Risk	Speed	Cost	Best For
Mobile 4G/5G	Very Low	Medium	$$$	Hardest targets
Residential Rotating	Low	Medium	$$	General scraping
Residential Static	Low	Fast	$$	Session-based work
ISP Proxies	Medium	Fast	$$	Light protection
Datacenter	BLOCKED	Fast	$	Non-protected sites only

Recommended Providers

Residential/Mobile Proxies

Bright Data (formerly Luminati) - Enterprise grade
Oxylabs - Good residential pool
Smartproxy - Budget friendly
IPRoyal - Cheap residential
PacketStream - P2P residential

Proxy Format

# Without auth
http://host:port
socks5://host:port

# With auth
http://username:password@host:port
socks5://username:password@host:port

# Rotating (some providers)
http://username:password@gate.provider.com:port

Configuration Examples

Camoufox

from camoufox.async_api import AsyncCamoufox

async with AsyncCamoufox(
    proxy={
        "server": "http://proxy.example.com:8080",
        "username": "user",
        "password": "pass"
    }
) as browser:
    page = await browser.new_page()

curl_cffi

from curl_cffi import requests

response = requests.get(
    "https://example.com",
    proxies={
        "http": "http://user:pass@proxy.example.com:8080",
        "https": "http://user:pass@proxy.example.com:8080"
    },
    impersonate="chrome120"
)

Environment Variable

export HTTP_PROXY="http://user:pass@proxy.example.com:8080"
export HTTPS_PROXY="http://user:pass@proxy.example.com:8080"

Session Stickiness

Don't rotate every request! This looks like a bot.

Real users:

Stay on same IP for 10-30 minutes
Make multiple requests from same session
Have consistent fingerprints

# Good: Sticky session
session_proxy = "http://user-session-abc123:pass@gate.provider.com:8080"

# Bad: Rotate every request
rotating_proxy = "http://user:pass@rotate.provider.com:8080"  # Only for initial requests

Geo-Targeting

Match proxy location to expected user location:

# Scraping US Airbnb listings? Use US proxy
us_proxy = "http://user-country-us:pass@gate.provider.com:8080"

# Scraping UK Yelp? Use UK proxy  
uk_proxy = "http://user-country-gb:pass@gate.provider.com:8080"

Testing Your Proxy

# Check IP and detection
distrobox-enter pybox -- python -c "
from curl_cffi import requests
r = requests.get('https://httpbin.org/ip', impersonate='chrome120', 
                 proxies={'https': 'http://user:pass@proxy:port'})
print(r.json())
"

# Check for bot detection
distrobox-enter pybox -- python scripts/nodriver-fetch.py \
    "https://bot.sannysoft.com" \
    --proxy "http://user:pass@proxy:port" \
    --screenshot bot-test.png

Cost Optimization

Use datacenter for non-protected sites - Save residential bandwidth
Cache aggressively - Don't re-fetch unchanged pages
Use API endpoints - Faster than full browser rendering
Rotate only when blocked - Start with sticky sessions
Off-peak scraping - Some providers charge less at night

Troubleshooting

Issue	Cause	Fix
403 immediately	Datacenter IP	Use residential
403 after few requests	IP burned	Rotate to new IP
Captcha appears	Suspicious behavior	Slow down, add randomness
Connection timeout	Bad proxy	Try different proxy/provider
SSL errors	Proxy doesn't support HTTPS	Use HTTPS proxy or tunnel

3.9 KiB Raw Blame History