Demote per-book and per-chapter-list-page Info logs to Debug — these
fire hundreds of times per catalogue run and drown out meaningful signals:
- orchestrator: RunBook starting (per book)
- metadata saved (per book)
- chapter list fetched (per book)
- scraping chapter list page N (per pagination page per book)
The 'book scrape finished' summary log (with scraped/skipped/errors
counters) remains at Info — it is the useful signal per book.
novelfire.net changed its book page structure. Old selectors produced empty
status and null genres for every book, causing all Meilisearch filters to
return zero results.
Old → new:
- status: <span class="status"> → <strong class="ongoing|completed|hiatus">
(text lowercased for consistent index values)
- genres: <div class="genres"> <a> → <div class="categories"> <a class="property-item">
(text lowercased for consistent index values)
Adds TestParseMetadataSelectors to guard against future regressions.
- scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) +
500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata
also switched to retryGet so a single 429 on a book page is retried rather
than aborting the whole refresh.
- catalogue_refresh.go: per-book delay is now configurable
(RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter
applied before every metadata fetch. Only metadata is scraped here —
chapters are fetched on-demand, not during catalogue refresh. Progress
logged every 50 books instead of 100.
- config.go / runner.go / main.go: add CatalogueRequestDelay field wired
from RUNNER_CATALOGUE_REQUEST_DELAY env var.
- release.yaml: comment out upload-sourcemaps job and remove it from the
release needs; GlitchTip auth token needs refreshing after DB wipe.
- backend/: Go API server and runner binaries with PocketBase + MinIO storage
- ui-v2/: SvelteKit frontend rewrite
- docker-compose-new.yml: compose file for the v2 stack
- .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images
- scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs
- .env.example: document DOCKER_BUILDKIT=1 for Colima users
