Welcome to TRAWL! 👋
Self-hosted web scraping engine with best-effort JS challenge and CAPTCHA solving.
Dedicated flows for Cloudflare, Akamai Bot Manager, and Imperva/Incapsula (best effort), plus Turnstile, reCAPTCHA, hCaptcha, and GeeTest.
Much faster and more reliable FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.
- 2-6x faster - compared to FlareSolverr or Byparr it returns much faster with higher success rate
- 4-tier execution - plain HTTP fetch → cached browser session → fresh challenge solve → residential proxy
- Challenge-aware HTTP/HTTPS proxy - direct forwarding for normal traffic, automatic tier escalation for detected walls, plus WebSockets, binary bodies, and Range/206 support
- Multi-WAF handling - dedicated Cloudflare, Akamai Bot Manager, and Imperva/Incapsula detection and browser flows
- Native captcha solving - CF Turnstile/Interstitial, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide
- Camoufox Firefox - fingerprint-patched at the C++/Juggler level to reduce automation signals
- Session cache - solved cookies and browser identity stored in Redis; accepted sessions can avoid a fresh solve
- FlareSolverr compatible - works with Prowlarr, Jackett, Sonarr, and the full *arr ecosystem out of the box
- No paid solver API required - reCAPTCHA audio can use Google's free STT endpoint or an optional local Whisper service
View/Collapse All
|
NodeMaven - The most reliable proxy provider with the Highest Quality IP on the market. Best solution for automation, web scraping, SEO research, and social media management. Why NodeMaven? • 99.9% uptime • Sticky sessions up to 7 days • IP filtering: all proxies have fraud score <97% • No KYC required • Cashback on traffic - burn GB and earn up to 10% back Special codes for TRAWL users: • TRAWL35 - 35% off to Mobile and Residential Proxies • TRAWL40 - 40% off to ISP (Static) Proxies |
|
Swiftproxy - A global residential proxy provider offering 80M+ real residential IPs across 190+ countries and regions. Built for web scraping, browser automation, AI workflows, SEO monitoring, and multi-account management. Why Swiftproxy? • Non-expiring residential traffic • Rotating & sticky sessions • HTTP, HTTPS & SOCKS5 support • Country, state & city-level targeting • 24/7 technical support Exclusive offer for TRAWL users: • PROXY90 - Get 10% OFF all proxy plans • Free Trial Available |
# Clone and configure
git clone https://github.com/germondai/trawl
cd trawl
cp .env.example .env
# Start scraper + Redis
docker compose up -d
# Verify
curl http://localhost:8191/healthFirst boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.
Prefer a one-click installation? TRAWL is available from the community app catalogs for both TrueNAS and Unraid:
- TrueNAS Community Apps — open Apps → Discover Apps and search for TRAWL.
- Unraid Community Apps — open the Apps tab and search for Trawl.
Thanks to the TrueNAS and Unraid community contributors who packaged and published these integrations.
curl -X POST http://localhost:8191/v1 \
-H 'Content-Type: application/json' \
-d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'Returns richer metadata: tier, timings, sessionCached, full cookie list.
curl -X POST http://localhost:8191/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"https://nowsecure.nl","maxTimeout":60000}'Set the FlareSolverr URL to:
http://localhost:8191 # running on the same host
http://trawl:8191 # running via Docker Compose on the same network
Some sites bind their Cloudflare clearance to the solving browser's full connection
fingerprint. The /v1 flow can't help there: Prowlarr keeps only the cookie + user-agent
and re-fetches the page with its own HTTP client, which Cloudflare re-challenges — the
cookie isn't portable. For those indexers, enable TRAWL's forward proxy and add it to
Prowlarr as an HTTP proxy:
MITM_PROXY_ENABLED=true
MITM_PROXY_PORT=8192
MITM_PROXY_CA_DIR=/data/proxy-ca # persist the CA (mount a volume)
MITM_PROXY_MAX_TIER=4 # cap escalation (e.g. 3 to stay off residential)By default the listener binds 0.0.0.0 so clients on a Docker bridge network can reach
it; set MITM_PROXY_HOST=127.0.0.1 to restrict it to loopback on a bare-metal host.
- Install the proxy's CA into the client's trust store so it accepts the per-host certs:
curl http://<trawl-host>:8191/proxy-ca.crt→ add to the Prowlarr container's CA store (e.g. a linuxserver/custom-cont-init.dscript that copies it to/usr/local/share/ca-certificates/and runsupdate-ca-certificates). - Prowlarr → Settings → Indexer Proxies → HTTP, host
<trawl-host>, port8192. Give it a tag if only selected indexers should use it.
Ordinary requests use a direct HTTP/TLS path. Small HTML, JSON, and text responses are buffered
for challenge detection; detected challenges escalate through the same tier pipeline as
POST /scrape. Videos and large binary responses stream directly. Range requests are forwarded
end to end and can escalate when their response is a detected challenge; WebSocket upgrades use a
direct relay without browser escalation.
See the complete proxy documentation for routing details, supported traffic, limitations, CA installation, and client examples.
⚠️ A MITM proxy can impersonate any host to a client that trusts its CA. Only expose it on a private interface (localhost / a private Docker network), never publicly.
The proxy self-generates a root CA on first run. Its certificate and private key are persisted
under MITM_PROXY_CA_DIR (default /data/proxy-ca). Per-host certificates are minted and cached
in memory while TRAWL runs; they do not need separate installation because they are signed by the
persistent root. Every client that uses the proxy must trust that root. Without it, HTTPS fails with
ERR_CERT_AUTHORITY_INVALID (browsers) or PKIX path building failed (Java).
Download the CA once per client:
curl http://<trawl-host>:8191/proxy-ca.crt -o trawl-ca.crt
# or in a Docker setup where the API isn't reachable from outside:
docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crtsudo security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain ./trawl-ca.crt
# Verify
security find-certificate -c "TRAWL MITM Proxy CA"
# Remove later
sudo security delete-certificate -c "TRAWL MITM Proxy CA" \
/Library/Keychains/System.keychainsudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
sudo update-ca-certificates
# Verify
awk '/BEGIN/{c++} c==2' /etc/ssl/certs/ca-certificates.crt | grep -c "TRAWL MITM"sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt
sudo update-ca-trustFirefox installations that do not use operating-system roots need a per-profile NSS import:
# Firefox 115+ uses a file-backed NSS DB; older versions use the legacy libnssdb format.
# The certutil command is the same either way.
certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \
-d sql:$HOME/.mozilla/firefox/<profile-dir>
# Or via Firefox UI: Settings → Privacy & Security → Certificates → View Certificates →
# Authorities → Import… → check "Trust this CA to identify websites".
# Profile dir location: about:profiles in Firefox.Chrome uses the system trust store on macOS and Windows but has its own on Linux:
# Option A: launch Chrome with --user-data-dir + NSS DB update (same as Firefox).
# Option B: use Chrome's --ignore-certificate-errors-spki-list=<hash> (per-session, less safe).
# Option C: add the cert to the system store (above) — Chrome picks it up automatically on
# most Linux distros via the nss-tool lookup.# Find the JRE cacerts file for your client.
# JDownloader: <install>/jre/lib/security/cacerts
keytool -importcert -alias trawl -file trawl-ca.crt \
-keystore "<path-to-cacerts>" -storepass changeit
# If `keytool` reports "Certificate already exists in keystore", use -delete first:
# keytool -delete -alias trawl -keystore "<path-to-cacerts>" -storepass changeitProwlarr, Sonarr, and Radarr are .NET applications, not Java applications. For their
Docker-based installations, add the CA to the container's Linux system trust store. A common
LinuxServer pattern is a /custom-cont-init.d script:
# In the client's Compose service:
volumes:
- ./trawl-ca.crt:/config/trawl-ca.crt:ro
- ./install-trawl-ca.sh:/custom-cont-init.d/50-install-trawl-ca:ro#!/usr/bin/with-contenv bash
cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
update-ca-certificatesLinuxServer runs scripts in /custom-cont-init.d/ when the container starts. Java clients such as
JDownloader require the separate keytool import described above.
JDownloader bundles its own JRE; the CA must be imported into it.
- Find the JRE:
Settings → Advanced → Java Path(in JDownloader) or look in the install dir:- Windows:
C:\Program Files\JDownloader 2\jre\lib\security\cacerts - macOS:
/Applications/JDownloader 2.app/Contents/app/jre/lib/security/cacerts - Linux:
<install>/jre/lib/security/cacerts
- Windows:
- Run the
keytool -importcertcommand above against that file. - Restart JDownloader.
# Run PowerShell as Administrator.
Import-Certificate -FilePath .\trawl-ca.crt `
-CertStoreLocation Cert:\LocalMachine\Root
# Remove later
Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*TRAWL MITM*" } | Remove-ItemEvery installation method has a symmetric removal path. Search your trust store for
TRAWL MITM Proxy CA (the CA's CN) and delete that entry. The CA certificate and key also live at
<MITM_PROXY_CA_DIR>/ca.crt and ca.key on the TRAWL host. Deleting either causes TRAWL to
generate a new root on its next start, so existing clients must install the new certificate.
Request
│
▼
Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms)
│ blocked
▼
Tier 2: Cached session ─────── success ──→ return (~500ms)
│ cache miss / expired
▼
Tier 3: Fresh challenge solve ─ success ──→ cache + return
│ IP flagged
▼
Tier 4: Residential proxy ──── success ──→ cache + return (15–45s)
│ failed
▼
error
| File | Description |
|---|---|
docker-compose.yml |
Scraper + Redis (default) |
docker-compose.minimal.yml |
Scraper only, no Redis |
docker-compose.prod.yml |
Production: restart: always, memory limit, healthcheck |
| Image tag | Built from | Runtime | Use case |
|---|---|---|---|
ghcr.io/germondai/trawl:latest |
apps/api/Dockerfile |
Bun 1.3.14 (modern, AVX2) | Default — modern Linux amd64/arm64 |
ghcr.io/germondai/trawl:baseline |
apps/api/Dockerfile.baseline |
Bun 1.3.14 baseline (no AVX2) | Older CPUs / older kernels (Synology NAS, J4125, Atom-era) |
Both tags live on the same ghcr.io/germondai/trawl package — they share the registry but use different Dockerfile sources. Pick whichever tag fits your hardware:
# Modern hardware (most users)
image: ghcr.io/germondai/trawl:latest
# Older CPUs without AVX2 / Synology / older kernels
image: ghcr.io/germondai/trawl:baselineSynology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The :baseline tag is published for that case — confirmed working on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, /health reports healthy, and it solves live Cloudflare challenges via /v1 (see #1). Published by independent GitHub Actions workflows (.github/workflows/publish.yml, publish-baseline.yml); tag-triggered releases push matching git tags (e.g. v1.0.0 → 1.0.0, 1.0.0-baseline → 1.0.0-baseline) and manual workflow_dispatch from main updates the rolling tag (latest and baseline respectively).
TRAWL follows Semantic Versioning. Pushing a v-prefixed git tag (e.g.
v1.0.0) triggers publish.yml, which builds and pushes the matching un-prefixed Docker tag
(ghcr.io/germondai/trawl:1.0.0) alongside a major-only tag (:1). :latest always tracks the
tip of main; :sha-<shortsha> images are pushed on every main commit regardless of tags.
See the Releases page for the full version
history and CHANGELOG.md for what changed in each one.
To publish a specific past commit that had a major fix without waiting for the next tip-of-main
release, tag that exact SHA and push it — publish.yml builds off the tag ref, not off main's
current tip:
git tag -a v1.0.1 <sha> -m "..."
git push origin v1.0.1TRAWL supports HTTP proxies, authenticated HTTP proxies, and SOCKS5 proxies. The standard Compose
files read proxy settings from the local .env file:
# Optional Tier 3 datacenter proxy
PROXY_URL=http://user:[email protected]:8080
# Optional Tier 4 residential proxy
RESIDENTIAL_PROXY_URL=socks5://user:[email protected]:1080docker compose up -dLeave either value empty to disable that proxy tier. Multiple endpoints can be separated with
commas; larger pools can use the corresponding *_LIST_FILE variable. See
Configuration → Proxies
for pool and mounted-file examples.
| Variable | Default | Description |
|---|---|---|
BROWSER_POOL_SIZE |
3 |
Warm Camoufox Firefox instances |
BROWSER_ACQUIRE_TIMEOUT_MS |
15000 |
How long acquire() polls for a free browser before HTTP 429 is returned |
BROWSER_RECYCLE_AFTER_CONTEXTS |
8 |
Recycle a browser after this many blocked/needs-js outcomes; set 0 to disable |
BROWSER_CONTENT_PROCESSES |
2 |
Cap Firefox content processes per browser (dom.ipc.processCount); lowers RAM/CPU |
SESSION_TTL_SECONDS |
3600 |
Redis session cache TTL (seconds) |
REDIS_URL |
redis://localhost:6379 |
Redis connection string |
PROXY_URL |
— | Optional Tier 3 HTTP or SOCKS5 proxy, or comma-separated pool |
PROXY_LIST_FILE |
— | File containing one Tier 3 proxy URL per line |
RESIDENTIAL_PROXY_URL |
— | Enables Tier 4 proxy escalation |
RESIDENTIAL_PROXY_LIST_FILE |
— | File containing one Tier 4 proxy URL per line |
STT_URL |
— | Local Whisper endpoint for reCAPTCHA (optional) |
PORT |
8191 |
API listen port |
MITM_PROXY_ENABLED |
false |
Enable the challenge-bypassing HTTP/HTTPS proxy |
MITM_PROXY_PORT |
8192 |
Forward-proxy listen port |
MITM_PROXY_HOST |
0.0.0.0 |
Bind address; 127.0.0.1 for loopback-only |
MITM_PROXY_CA_DIR |
/data/proxy-ca |
Persistent root CA certificate and private-key directory |
MITM_PROXY_MAX_TIER |
4 |
Cap escalation used by the proxy (e.g. 3 to stay off residential) |
MITM_PROXY_DEBUG |
false |
Log one line per proxied request (errors are always logged) |
Built on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching, Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy Node/Express baggage.
| Layer | Technology |
|---|---|
| Runtime | Bun |
| API | Elysia |
| Browser | Camoufox Firefox (via camoufox-js) |
| Session cache | Redis 8.8 |
| Landing page | Nuxt 4 |
| Documentation | VitePress |
Made with ❤️ by @germondai
