146 lines
3.9 KiB
Markdown
146 lines
3.9 KiB
Markdown
# Proxy Setup Guide
|
|
|
|
## Why Proxies Matter
|
|
|
|
**Datacenter IPs = INSTANT BLOCK** on protected sites like Airbnb, Yelp, and anything with Cloudflare/Datadome.
|
|
|
|
Anti-bot systems check IP reputation scores. AWS, DigitalOcean, Vultr, Linode IPs have high "fraud scores" and are blocked immediately.
|
|
|
|
## Proxy Types (Best to Worst for Anti-Bot)
|
|
|
|
| Type | Detection Risk | Speed | Cost | Best For |
|
|
|------|---------------|-------|------|----------|
|
|
| **Mobile 4G/5G** | Very Low | Medium | $$$ | Hardest targets |
|
|
| **Residential Rotating** | Low | Medium | $$ | General scraping |
|
|
| **Residential Static** | Low | Fast | $$ | Session-based work |
|
|
| **ISP Proxies** | Medium | Fast | $$ | Light protection |
|
|
| **Datacenter** | BLOCKED | Fast | $ | Non-protected sites only |
|
|
|
|
## Recommended Providers
|
|
|
|
### Residential/Mobile Proxies
|
|
- **Bright Data** (formerly Luminati) - Enterprise grade
|
|
- **Oxylabs** - Good residential pool
|
|
- **Smartproxy** - Budget friendly
|
|
- **IPRoyal** - Cheap residential
|
|
- **PacketStream** - P2P residential
|
|
|
|
### Proxy Format
|
|
|
|
```
|
|
# Without auth
|
|
http://host:port
|
|
socks5://host:port
|
|
|
|
# With auth
|
|
http://username:password@host:port
|
|
socks5://username:password@host:port
|
|
|
|
# Rotating (some providers)
|
|
http://username:password@gate.provider.com:port
|
|
```
|
|
|
|
## Configuration Examples
|
|
|
|
### Camoufox
|
|
|
|
```python
|
|
from camoufox.async_api import AsyncCamoufox
|
|
|
|
async with AsyncCamoufox(
|
|
proxy={
|
|
"server": "http://proxy.example.com:8080",
|
|
"username": "user",
|
|
"password": "pass"
|
|
}
|
|
) as browser:
|
|
page = await browser.new_page()
|
|
```
|
|
|
|
### curl_cffi
|
|
|
|
```python
|
|
from curl_cffi import requests
|
|
|
|
response = requests.get(
|
|
"https://example.com",
|
|
proxies={
|
|
"http": "http://user:pass@proxy.example.com:8080",
|
|
"https": "http://user:pass@proxy.example.com:8080"
|
|
},
|
|
impersonate="chrome120"
|
|
)
|
|
```
|
|
|
|
### Environment Variable
|
|
|
|
```bash
|
|
export HTTP_PROXY="http://user:pass@proxy.example.com:8080"
|
|
export HTTPS_PROXY="http://user:pass@proxy.example.com:8080"
|
|
```
|
|
|
|
## Session Stickiness
|
|
|
|
**Don't rotate every request!** This looks like a bot.
|
|
|
|
Real users:
|
|
- Stay on same IP for 10-30 minutes
|
|
- Make multiple requests from same session
|
|
- Have consistent fingerprints
|
|
|
|
```python
|
|
# Good: Sticky session
|
|
session_proxy = "http://user-session-abc123:pass@gate.provider.com:8080"
|
|
|
|
# Bad: Rotate every request
|
|
rotating_proxy = "http://user:pass@rotate.provider.com:8080" # Only for initial requests
|
|
```
|
|
|
|
## Geo-Targeting
|
|
|
|
Match proxy location to expected user location:
|
|
|
|
```python
|
|
# Scraping US Airbnb listings? Use US proxy
|
|
us_proxy = "http://user-country-us:pass@gate.provider.com:8080"
|
|
|
|
# Scraping UK Yelp? Use UK proxy
|
|
uk_proxy = "http://user-country-gb:pass@gate.provider.com:8080"
|
|
```
|
|
|
|
## Testing Your Proxy
|
|
|
|
```bash
|
|
# Check IP and detection
|
|
distrobox-enter pybox -- python -c "
|
|
from curl_cffi import requests
|
|
r = requests.get('https://httpbin.org/ip', impersonate='chrome120',
|
|
proxies={'https': 'http://user:pass@proxy:port'})
|
|
print(r.json())
|
|
"
|
|
|
|
# Check for bot detection
|
|
distrobox-enter pybox -- python scripts/nodriver-fetch.py \
|
|
"https://bot.sannysoft.com" \
|
|
--proxy "http://user:pass@proxy:port" \
|
|
--screenshot bot-test.png
|
|
```
|
|
|
|
## Cost Optimization
|
|
|
|
1. **Use datacenter for non-protected sites** - Save residential bandwidth
|
|
2. **Cache aggressively** - Don't re-fetch unchanged pages
|
|
3. **Use API endpoints** - Faster than full browser rendering
|
|
4. **Rotate only when blocked** - Start with sticky sessions
|
|
5. **Off-peak scraping** - Some providers charge less at night
|
|
|
|
## Troubleshooting
|
|
|
|
| Issue | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| 403 immediately | Datacenter IP | Use residential |
|
|
| 403 after few requests | IP burned | Rotate to new IP |
|
|
| Captcha appears | Suspicious behavior | Slow down, add randomness |
|
|
| Connection timeout | Bad proxy | Try different proxy/provider |
|
|
| SSL errors | Proxy doesn't support HTTPS | Use HTTPS proxy or tunnel |
|