feat(scraper): harden browser headers and add proxy support
All checks were successful
CI / Scraper / Lint (push) Successful in 8s
CI / Scraper / Lint (pull_request) Successful in 12s
CI / Scraper / Test (pull_request) Successful in 8s
CI / Scraper / Test (push) Successful in 16s
Release / Scraper / Test (push) Successful in 8s
Release / UI / Build (push) Successful in 24s
CI / Scraper / Docker Push (pull_request) Has been skipped
CI / UI / Build (pull_request) Successful in 30s
CI / Scraper / Docker Push (push) Successful in 39s
CI / UI / Docker Push (pull_request) Has been skipped
Release / UI / Docker (push) Successful in 40s
Release / Scraper / Docker (push) Successful in 50s
iOS CI / Build (pull_request) Successful in 3m7s
iOS CI / Test (pull_request) Successful in 5m23s

Upgrade DirectHTTPClient to send a full Chrome 124 header set
(Sec-Fetch-*, Accept-Encoding with gzip decompression, Referer,
Cache-Control) to reduce bot-detection false positives on WAFs.

Add SCRAPER_PROXY env var to route all outbound scrape requests
through a configurable proxy (residential or otherwise); falls back
to the standard HTTP_PROXY / HTTPS_PROXY env vars.
This commit is contained in:
Admin
2026-03-14 18:40:32 +05:00
parent 02705dc6ed
commit 1642434a79
2 changed files with 78 additions and 8 deletions

View File

@@ -12,6 +12,10 @@
//
// SCRAPER_WORKERS Chapter goroutine count (default: NumCPU)
// SCRAPER_HTTP_ADDR HTTP listen address (default: :8080)
// SCRAPER_PROXY Outbound proxy for all scraper requests, e.g.
// http://user:pass@proxy-host:3128 — use a
// residential proxy to bypass datacenter IP blocks.
// Falls back to HTTP_PROXY / HTTPS_PROXY if unset.
// KOKORO_URL Kokoro-FastAPI base URL (default: "")
// KOKORO_VOICE Default TTS voice (default: af_bella)
// POCKETBASE_URL PocketBase API base URL (default: http://localhost:8090)
@@ -493,6 +497,7 @@ Environment variables:
SCRAPER_WORKERS Chapter goroutines (default: NumCPU = %d)
SCRAPER_HTTP_ADDR HTTP listen address (default: :8080)
SCRAPER_TIMEOUT HTTP request timeout sec (default: 90)
SCRAPER_PROXY Outbound proxy URL (default: "", falls back to HTTP_PROXY/HTTPS_PROXY)
KOKORO_URL Kokoro-FastAPI base URL (default: "", TTS disabled)
KOKORO_VOICE Default TTS voice (default: af_bella)
POCKETBASE_URL PocketBase base URL (default: http://localhost:8090)