kamil/libnovel
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 162 Wiki Activity

perf(scraper): switch chapter list to direct HTTP, remove Browserless dependency

Browse Source 
novelfire.net chapter-list pages (/chapters?page=N) are server-rendered —
verified via curl. Switch urlClient to NewDirectHTTPClient alongside the
existing chapterClient. Remove BROWSERLESS_URL_STRATEGY env var and clean
up the now-irrelevant WaitFor/GotoOptions fields from both ScrapeChapterList
and ScrapeChapterListPage ContentRequests.
This commit is contained in:
Admin
2026-03-03 20:46:22 +05:00
parent cff0c78b4f
commit cfd893d24b
4 changed files with 8 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docker-compose.yml
View File

@@ -119,10 +119,9 @@ services:
environment:
BROWSERLESS_URL: "http://browserless:${BROWSERLESS_PORT:-3030}"
BROWSERLESS_TOKEN: "${BROWSERLESS_TOKEN:-}"
# content | scrape | cdp | direct — swap to test different strategies.
# content | scrape | cdp | direct — used for catalogue, metadata, ranking.
# Chapter list and chapter text always use direct HTTP regardless of this setting.
BROWSERLESS_STRATEGY: "${BROWSERLESS_STRATEGY:-direct}"
# Strategy for URL retrieval (chapter list). Default: content (browserless)
BROWSERLESS_URL_STRATEGY: "${BROWSERLESS_URL_STRATEGY:-content}"
# 0 → defaults to NumCPU inside the container.
SCRAPER_WORKERS: "${SCRAPER_WORKERS:-0}"
SCRAPER_STATIC_ROOT: "/app/static/books"