每日备份 2026-06-10 12:57:13
This commit is contained in:
root-backup committed
1 parent
fda2ccb8c2
commit
3d2e638c6a
244 files changed
+58890
-3
No files matched your search
@@ -0,0 +1,72 @@
|
||||
# Camoufox Stealth Browser 🦊
|
||||
|
||||
**C++ level** anti-bot evasion — not JavaScript band-aids.
|
||||
|
||||
[](LICENSE)
|
||||
|
||||
## Why Camoufox > Chrome-Based Tools
|
||||
|
||||
| Approach | Patches At | Detectable? |
|
||||
|----------|-----------|-------------|
|
||||
| **Camoufox** ✅ | C++ (compiled into browser) | No — fingerprints are genuinely different |
|
||||
| undetected-chromedriver | JS runtime | Yes — timing analysis reveals patches |
|
||||
| puppeteer-stealth | JS injection | Yes — applied after page load |
|
||||
| playwright-stealth | JS injection | Yes — same limitations |
|
||||
|
||||
Most "stealth" tools patch Chrome with JavaScript after the browser starts. Anti-bot systems detect this via timing analysis and consistency checks.
|
||||
|
||||
**Camoufox is different.** It's a Firefox fork with stealth patches compiled into the C++ source code. WebGL, Canvas, and AudioContext fingerprints are genuinely spoofed — not masked by JS overrides.
|
||||
|
||||
## Key Features
|
||||
|
||||
- 🦊 **C++ Level Stealth** — Fingerprints baked into the browser binary
|
||||
- 📦 **Container Isolation** — Runs in distrobox, keeps host clean
|
||||
- ⚡ **Dual-Tool Design** — Camoufox for browsers, curl_cffi for fast API-only scraping
|
||||
- 🔥 **Firefox-Based** — Less fingerprinted than Chrome (bots love Chrome)
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Setup (first time)
|
||||
distrobox-enter pybox -- python3.14 -m pip install camoufox curl_cffi
|
||||
|
||||
# Fetch a Cloudflare-protected page
|
||||
distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \
|
||||
"https://yelp.com/biz/example" --headless
|
||||
|
||||
# API scraping (no browser needed)
|
||||
distrobox-enter pybox -- python3.14 scripts/curl-api.py \
|
||||
"https://api.example.com" --impersonate chrome120
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- `distrobox` with a `pybox` container
|
||||
- Residential proxy for Airbnb/Yelp (datacenter IPs = instant block)
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Use Case | Speed |
|
||||
|------|----------|-------|
|
||||
| **Camoufox** | Full browser automation, JS-heavy sites | ~3-5s/page |
|
||||
| **curl_cffi** | API endpoints, no JS needed | ~100ms/request |
|
||||
|
||||
## Documentation
|
||||
|
||||
- [SKILL.md](SKILL.md) — Full usage guide with session management
|
||||
- [references/proxy-setup.md](references/proxy-setup.md) — Proxy configuration
|
||||
- [references/fingerprint-checks.md](references/fingerprint-checks.md) — What anti-bot systems check
|
||||
|
||||
## Comparison with Other Skills
|
||||
|
||||
This skill focuses on **doing one thing well**: C++ level stealth with Camoufox.
|
||||
|
||||
For CAPTCHA solving, task checkpointing, and proxy rotation, see the [GitHub issues](https://github.com/kesslerio/stealth-browser-clawhub-skill/issues) for planned features.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 — See [LICENSE](LICENSE)
|
||||
|
||||
---
|
||||
|
||||
Made with 🦊 by [Kessler.io](https://kessler.io)
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: camoufox
|
||||
version: 1.0.0
|
||||
description: Anti-detect browser automation using Camoufox (Firefox-based). Use instead of Chrome/Playwright for bot-detection-heavy sites like X/Twitter, Naver, and other protected sites. Provides stealth browsing with OS/browser fingerprint spoofing, humanized mouse movements, and persistent sessions.
|
||||
---
|
||||
|
||||
# Camoufox
|
||||
|
||||
**Use instead of OpenClaw's built-in browser tool for bot-detection sites.**
|
||||
|
||||
## Setup (one-time)
|
||||
```bash
|
||||
bash scripts/setup.sh
|
||||
```
|
||||
|
||||
## Usage
|
||||
```bash
|
||||
source ~/.openclaw/workspace/camoufox-env/bin/activate
|
||||
xvfb-run -a --server-args="-screen 0 1920x1080x24" python3 your_script.py
|
||||
```
|
||||
|
||||
## Required Settings
|
||||
```python
|
||||
from camoufox.sync_api import Camoufox
|
||||
|
||||
with Camoufox(
|
||||
headless=False, # MUST be False (True gets detected)
|
||||
os='linux', # MUST match server OS (stack traces leak real OS)
|
||||
humanize=True,
|
||||
persistent_context=True,
|
||||
user_data_dir='~/.openclaw/camoufox-profile'
|
||||
) as browser:
|
||||
page = browser.new_page()
|
||||
# Use standard Playwright API from here
|
||||
```
|
||||
|
||||
## Paths
|
||||
- venv: `~/.openclaw/workspace/camoufox-env/`
|
||||
- profile: `~/.openclaw/camoufox-profile/`
|
||||
|
||||
## CAPTCHA Handling
|
||||
First visit to protected sites may need manual CAPTCHA:
|
||||
```bash
|
||||
DISPLAY=:1 python3 scripts/login_session.py https://site.com/login
|
||||
```
|
||||
Solve via VNC, session saves to profile for future automation.
|
||||
|
||||
## Scripts
|
||||
- `scripts/setup.sh` - Install dependencies
|
||||
- `scripts/browse.py <url> --screenshot out.png` - Quick browse
|
||||
- `scripts/login_session.py <url>` - VNC manual login
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"owner": "kesslerio",
|
||||
"slug": "camoufox-stealth",
|
||||
"displayName": "Camoufox Stealth Browser",
|
||||
"latest": {
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1770018858396,
|
||||
"commit": "https://github.com/clawdbot/skills/commit/d9bc4033c40e02a8cfd481bf4a363bbe072104a4"
|
||||
},
|
||||
"history": []
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
# Browser Fingerprint Checks
|
||||
|
||||
What anti-bot systems check and how to evade.
|
||||
|
||||
## Detection Layers
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Layer 1: Network/IP │
|
||||
│ • IP reputation score │
|
||||
│ • Datacenter vs residential │
|
||||
│ • Geo-location consistency │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Layer 2: TLS Fingerprint │
|
||||
│ • JA3/JA4 hash │
|
||||
│ • Cipher suite order │
|
||||
│ • TLS extensions │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Layer 3: HTTP Headers │
|
||||
│ • User-Agent consistency │
|
||||
│ • Header order │
|
||||
│ • Accept-Language │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Layer 4: Browser Fingerprint │
|
||||
│ • navigator.webdriver │
|
||||
│ • Canvas/WebGL hash │
|
||||
│ • Audio fingerprint │
|
||||
│ • Font enumeration │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Layer 5: Behavioral │
|
||||
│ • Mouse movements │
|
||||
│ • Scroll patterns │
|
||||
│ • Timing analysis │
|
||||
│ • Click patterns │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Specific Checks & Countermeasures
|
||||
|
||||
### 1. navigator.webdriver
|
||||
|
||||
**What:** `navigator.webdriver` returns `true` for automated browsers.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
navigator.webdriver // true = bot
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Nodriver: Automatically patches
|
||||
- Camoufox: Patched at C++ level
|
||||
- Playwright-stealth: `delete navigator.webdriver` (weak, can be detected)
|
||||
|
||||
### 2. Chrome DevTools Protocol (CDP)
|
||||
|
||||
**What:** Presence of CDP side-channels reveals automation.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
// Sites check for CDP artifacts
|
||||
window.cdc_adoQpoasnfa76pfcZLmcfl_Array
|
||||
window.cdc_adoQpoasnfa76pfcZLmcfl_Promise
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Nodriver: Uses CDP but patches artifacts
|
||||
- Camoufox: No CDP (native Firefox)
|
||||
- Patchright: Patches CDP artifacts
|
||||
|
||||
### 3. Canvas Fingerprint
|
||||
|
||||
**What:** Each GPU/driver combo renders slightly differently.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
canvas.toDataURL() // Unique hash per system
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Camoufox: Injects deterministic noise
|
||||
- Some detection tools look for perfectly identical canvas = bot
|
||||
|
||||
### 4. WebGL Fingerprint
|
||||
|
||||
**What:** GPU vendor/renderer strings.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
gl.getParameter(gl.RENDERER) // "ANGLE (Intel HD Graphics)"
|
||||
gl.getParameter(gl.VENDOR) // "Google Inc."
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Camoufox: Spoofs to common values
|
||||
- Avoid VMs with "llvmpipe" or "SwiftShader" (instant bot flag)
|
||||
|
||||
### 5. Audio Fingerprint
|
||||
|
||||
**What:** AudioContext processing variations.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
new AudioContext().createOscillator() // Unique waveform
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Camoufox: Adds noise to audio processing
|
||||
|
||||
### 6. Font Enumeration
|
||||
|
||||
**What:** Bots often have minimal fonts installed.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
// Test if specific fonts render
|
||||
document.fonts.check('12px "Segoe UI"')
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Install standard fonts in your environment
|
||||
- Camoufox: Spoofs font list
|
||||
|
||||
### 7. Screen/Window Size
|
||||
|
||||
**What:** Unusual dimensions = bot.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
screen.width, screen.height
|
||||
window.innerWidth, window.innerHeight
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Use standard resolutions: 1920x1080, 1366x768
|
||||
- Avoid: 800x600, odd dimensions
|
||||
|
||||
### 8. Timezone Consistency
|
||||
|
||||
**What:** Timezone must match IP geo-location.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
Intl.DateTimeFormat().resolvedOptions().timeZone
|
||||
new Date().getTimezoneOffset()
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Set timezone to match proxy location:
|
||||
```bash
|
||||
TZ="America/New_York" python script.py
|
||||
```
|
||||
|
||||
### 9. Language Consistency
|
||||
|
||||
**What:** Browser language should match geo.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
navigator.language // "en-US"
|
||||
navigator.languages // ["en-US", "en"]
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Configure browser locale to match proxy:
|
||||
```python
|
||||
# Camoufox
|
||||
async with AsyncCamoufox(locale="en-US") as browser:
|
||||
```
|
||||
|
||||
### 10. Hardware Concurrency
|
||||
|
||||
**What:** VMs often report 1-2 CPUs.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
navigator.hardwareConcurrency // 1 or 2 = suspicious
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Camoufox: Spoofs to 4-8
|
||||
- Or use real hardware, not minimal VM
|
||||
|
||||
### 11. Headless Detection
|
||||
|
||||
**What:** Multiple signals reveal headless mode.
|
||||
|
||||
**Checks:**
|
||||
```javascript
|
||||
navigator.plugins.length === 0
|
||||
navigator.mimeTypes.length === 0
|
||||
window.chrome === undefined
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Use "new headless" mode: `headless="new"`
|
||||
- Or run headed in Xvfb
|
||||
- Camoufox headless is well-patched
|
||||
|
||||
### 12. Mouse/Keyboard Events
|
||||
|
||||
**What:** Bots often have no mouse history.
|
||||
|
||||
**Check:**
|
||||
```javascript
|
||||
// Sites track mouse movement patterns
|
||||
document.addEventListener('mousemove', analyze)
|
||||
```
|
||||
|
||||
**Countermeasure:**
|
||||
- Inject random mouse movements
|
||||
- Nodriver/Camoufox: Built-in human-like movement
|
||||
- Don't teleport cursor - use bezier curves
|
||||
|
||||
## Test Sites
|
||||
|
||||
Check your stealth setup:
|
||||
|
||||
| Site | What It Tests |
|
||||
|------|--------------|
|
||||
| https://bot.sannysoft.com | Comprehensive bot detection |
|
||||
| https://browserleaks.com | All fingerprints |
|
||||
| https://nowsecure.nl | Cloudflare detection |
|
||||
| https://pixelscan.net | Bot detection score |
|
||||
| https://abrahamjuliot.github.io/creepjs | Advanced fingerprinting |
|
||||
|
||||
## Tool Comparison
|
||||
|
||||
| Check | Camoufox | Plain Playwright |
|
||||
|-------|----------|------------------|
|
||||
| webdriver | ✅ | ❌ |
|
||||
| CDP artifacts | ✅ (no CDP) | ❌ |
|
||||
| Canvas | ✅ | ❌ |
|
||||
| WebGL | ✅ | ❌ |
|
||||
| Audio | ✅ | ❌ |
|
||||
| Fonts | ✅ | ❌ |
|
||||
| Headless | ✅ | ❌ |
|
||||
| TLS | ✅ | ❌ |
|
||||
|
||||
Legend: ✅ = patched, ❌ = detected
|
||||
@@ -0,0 +1,145 @@
|
||||
# Proxy Setup Guide
|
||||
|
||||
## Why Proxies Matter
|
||||
|
||||
**Datacenter IPs = INSTANT BLOCK** on protected sites like Airbnb, Yelp, and anything with Cloudflare/Datadome.
|
||||
|
||||
Anti-bot systems check IP reputation scores. AWS, DigitalOcean, Vultr, Linode IPs have high "fraud scores" and are blocked immediately.
|
||||
|
||||
## Proxy Types (Best to Worst for Anti-Bot)
|
||||
|
||||
| Type | Detection Risk | Speed | Cost | Best For |
|
||||
|------|---------------|-------|------|----------|
|
||||
| **Mobile 4G/5G** | Very Low | Medium | $$$ | Hardest targets |
|
||||
| **Residential Rotating** | Low | Medium | $$ | General scraping |
|
||||
| **Residential Static** | Low | Fast | $$ | Session-based work |
|
||||
| **ISP Proxies** | Medium | Fast | $$ | Light protection |
|
||||
| **Datacenter** | BLOCKED | Fast | $ | Non-protected sites only |
|
||||
|
||||
## Recommended Providers
|
||||
|
||||
### Residential/Mobile Proxies
|
||||
- **Bright Data** (formerly Luminati) - Enterprise grade
|
||||
- **Oxylabs** - Good residential pool
|
||||
- **Smartproxy** - Budget friendly
|
||||
- **IPRoyal** - Cheap residential
|
||||
- **PacketStream** - P2P residential
|
||||
|
||||
### Proxy Format
|
||||
|
||||
```
|
||||
# Without auth
|
||||
http://host:port
|
||||
socks5://host:port
|
||||
|
||||
# With auth
|
||||
http://username:password@host:port
|
||||
socks5://username:password@host:port
|
||||
|
||||
# Rotating (some providers)
|
||||
http://username:password@gate.provider.com:port
|
||||
```
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
### Camoufox
|
||||
|
||||
```python
|
||||
from camoufox.async_api import AsyncCamoufox
|
||||
|
||||
async with AsyncCamoufox(
|
||||
proxy={
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
) as browser:
|
||||
page = await browser.new_page()
|
||||
```
|
||||
|
||||
### curl_cffi
|
||||
|
||||
```python
|
||||
from curl_cffi import requests
|
||||
|
||||
response = requests.get(
|
||||
"https://example.com",
|
||||
proxies={
|
||||
"http": "http://user:pass@proxy.example.com:8080",
|
||||
"https": "http://user:pass@proxy.example.com:8080"
|
||||
},
|
||||
impersonate="chrome120"
|
||||
)
|
||||
```
|
||||
|
||||
### Environment Variable
|
||||
|
||||
```bash
|
||||
export HTTP_PROXY="http://user:pass@proxy.example.com:8080"
|
||||
export HTTPS_PROXY="http://user:pass@proxy.example.com:8080"
|
||||
```
|
||||
|
||||
## Session Stickiness
|
||||
|
||||
**Don't rotate every request!** This looks like a bot.
|
||||
|
||||
Real users:
|
||||
- Stay on same IP for 10-30 minutes
|
||||
- Make multiple requests from same session
|
||||
- Have consistent fingerprints
|
||||
|
||||
```python
|
||||
# Good: Sticky session
|
||||
session_proxy = "http://user-session-abc123:pass@gate.provider.com:8080"
|
||||
|
||||
# Bad: Rotate every request
|
||||
rotating_proxy = "http://user:pass@rotate.provider.com:8080" # Only for initial requests
|
||||
```
|
||||
|
||||
## Geo-Targeting
|
||||
|
||||
Match proxy location to expected user location:
|
||||
|
||||
```python
|
||||
# Scraping US Airbnb listings? Use US proxy
|
||||
us_proxy = "http://user-country-us:pass@gate.provider.com:8080"
|
||||
|
||||
# Scraping UK Yelp? Use UK proxy
|
||||
uk_proxy = "http://user-country-gb:pass@gate.provider.com:8080"
|
||||
```
|
||||
|
||||
## Testing Your Proxy
|
||||
|
||||
```bash
|
||||
# Check IP and detection
|
||||
distrobox-enter pybox -- python -c "
|
||||
from curl_cffi import requests
|
||||
r = requests.get('https://httpbin.org/ip', impersonate='chrome120',
|
||||
proxies={'https': 'http://user:pass@proxy:port'})
|
||||
print(r.json())
|
||||
"
|
||||
|
||||
# Check for bot detection
|
||||
distrobox-enter pybox -- python scripts/nodriver-fetch.py \
|
||||
"https://bot.sannysoft.com" \
|
||||
--proxy "http://user:pass@proxy:port" \
|
||||
--screenshot bot-test.png
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
1. **Use datacenter for non-protected sites** - Save residential bandwidth
|
||||
2. **Cache aggressively** - Don't re-fetch unchanged pages
|
||||
3. **Use API endpoints** - Faster than full browser rendering
|
||||
4. **Rotate only when blocked** - Start with sticky sessions
|
||||
5. **Off-peak scraping** - Some providers charge less at night
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| 403 immediately | Datacenter IP | Use residential |
|
||||
| 403 after few requests | IP burned | Rotate to new IP |
|
||||
| Captcha appears | Suspicious behavior | Slow down, add randomness |
|
||||
| Connection timeout | Bad proxy | Try different proxy/provider |
|
||||
| SSL errors | Proxy doesn't support HTTPS | Use HTTPS proxy or tunnel |
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
# Setup stealth browser tools in pybox
|
||||
# Usage: bash scripts/setup.sh
|
||||
|
||||
set -e
|
||||
|
||||
echo "🥷 Setting up stealth browser tools in pybox..."
|
||||
|
||||
# Check if distrobox is available
|
||||
if ! command -v distrobox &> /dev/null; then
|
||||
echo "❌ Error: distrobox not found. Install it first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if pybox exists
|
||||
if ! distrobox list | grep -q "pybox"; then
|
||||
echo "❌ Error: pybox container not found."
|
||||
echo " Create it with: distrobox create --name pybox --image fedora:latest"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📦 Installing Python packages..."
|
||||
distrobox-enter pybox -- pip install --upgrade pip
|
||||
distrobox-enter pybox -- pip install camoufox curl_cffi
|
||||
|
||||
echo "🦊 Installing Camoufox browser..."
|
||||
distrobox-enter pybox -- python -c "import camoufox; camoufox.install()"
|
||||
|
||||
echo "🔧 Installing system dependencies for headed mode..."
|
||||
distrobox-enter pybox -- sudo dnf install -y gtk3 libXt nss at-spi2-atk cups-libs libdrm mesa-libgbm 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "✅ Setup complete!"
|
||||
echo ""
|
||||
echo "Test with:"
|
||||
echo " distrobox-enter pybox -- python scripts/nodriver-fetch.py https://bot.sannysoft.com --screenshot test.png"
|
||||
echo ""
|
||||
echo "Or for maximum stealth:"
|
||||
echo " distrobox-enter pybox -- python scripts/camoufox-fetch.py https://nowsecure.nl --screenshot test.png"
|
||||
Reference in new issue
Block a user