feat(scraper): harden browser headers and add proxy support
All checks were successful
CI / Scraper / Lint (push) Successful in 8s
CI / Scraper / Lint (pull_request) Successful in 12s
CI / Scraper / Test (pull_request) Successful in 8s
CI / Scraper / Test (push) Successful in 16s
Release / Scraper / Test (push) Successful in 8s
Release / UI / Build (push) Successful in 24s
CI / Scraper / Docker Push (pull_request) Has been skipped
CI / UI / Build (pull_request) Successful in 30s
CI / Scraper / Docker Push (push) Successful in 39s
CI / UI / Docker Push (pull_request) Has been skipped
Release / UI / Docker (push) Successful in 40s
Release / Scraper / Docker (push) Successful in 50s
iOS CI / Build (pull_request) Successful in 3m7s
iOS CI / Test (pull_request) Successful in 5m23s
All checks were successful
CI / Scraper / Lint (push) Successful in 8s
CI / Scraper / Lint (pull_request) Successful in 12s
CI / Scraper / Test (pull_request) Successful in 8s
CI / Scraper / Test (push) Successful in 16s
Release / Scraper / Test (push) Successful in 8s
Release / UI / Build (push) Successful in 24s
CI / Scraper / Docker Push (pull_request) Has been skipped
CI / UI / Build (pull_request) Successful in 30s
CI / Scraper / Docker Push (push) Successful in 39s
CI / UI / Docker Push (pull_request) Has been skipped
Release / UI / Docker (push) Successful in 40s
Release / Scraper / Docker (push) Successful in 50s
iOS CI / Build (pull_request) Successful in 3m7s
iOS CI / Test (pull_request) Successful in 5m23s
Upgrade DirectHTTPClient to send a full Chrome 124 header set (Sec-Fetch-*, Accept-Encoding with gzip decompression, Referer, Cache-Control) to reduce bot-detection false positives on WAFs. Add SCRAPER_PROXY env var to route all outbound scrape requests through a configurable proxy (residential or otherwise); falls back to the standard HTTP_PROXY / HTTPS_PROXY env vars.
This commit is contained in:
@@ -12,6 +12,10 @@
|
||||
//
|
||||
// SCRAPER_WORKERS Chapter goroutine count (default: NumCPU)
|
||||
// SCRAPER_HTTP_ADDR HTTP listen address (default: :8080)
|
||||
// SCRAPER_PROXY Outbound proxy for all scraper requests, e.g.
|
||||
// http://user:pass@proxy-host:3128 — use a
|
||||
// residential proxy to bypass datacenter IP blocks.
|
||||
// Falls back to HTTP_PROXY / HTTPS_PROXY if unset.
|
||||
// KOKORO_URL Kokoro-FastAPI base URL (default: "")
|
||||
// KOKORO_VOICE Default TTS voice (default: af_bella)
|
||||
// POCKETBASE_URL PocketBase API base URL (default: http://localhost:8090)
|
||||
@@ -493,6 +497,7 @@ Environment variables:
|
||||
SCRAPER_WORKERS Chapter goroutines (default: NumCPU = %d)
|
||||
SCRAPER_HTTP_ADDR HTTP listen address (default: :8080)
|
||||
SCRAPER_TIMEOUT HTTP request timeout sec (default: 90)
|
||||
SCRAPER_PROXY Outbound proxy URL (default: "", falls back to HTTP_PROXY/HTTPS_PROXY)
|
||||
KOKORO_URL Kokoro-FastAPI base URL (default: "", TTS disabled)
|
||||
KOKORO_VOICE Default TTS voice (default: af_bella)
|
||||
POCKETBASE_URL PocketBase base URL (default: http://localhost:8090)
|
||||
|
||||
Reference in New Issue
Block a user