novelfire.net changed its book page structure. Old selectors produced empty
status and null genres for every book, causing all Meilisearch filters to
return zero results.
Old → new:
- status: <span class="status"> → <strong class="ongoing|completed|hiatus">
(text lowercased for consistent index values)
- genres: <div class="genres"> <a> → <div class="categories"> <a class="property-item">
(text lowercased for consistent index values)
Adds TestParseMetadataSelectors to guard against future regressions.
- scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) +
500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata
also switched to retryGet so a single 429 on a book page is retried rather
than aborting the whole refresh.
- catalogue_refresh.go: per-book delay is now configurable
(RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter
applied before every metadata fetch. Only metadata is scraped here —
chapters are fetched on-demand, not during catalogue refresh. Progress
logged every 50 books instead of 100.
- config.go / runner.go / main.go: add CatalogueRequestDelay field wired
from RUNNER_CATALOGUE_REQUEST_DELAY env var.
- release.yaml: comment out upload-sourcemaps job and remove it from the
release needs; GlitchTip auth token needs refreshing after DB wipe.
- backend/: Go API server and runner binaries with PocketBase + MinIO storage
- ui-v2/: SvelteKit frontend rewrite
- docker-compose-new.yml: compose file for the v2 stack
- .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images
- scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs
- .env.example: document DOCKER_BUILDKIT=1 for Colima users
