libnovel/AGENTS.md

libnovel Project

Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI. Structured data in PocketBase, binary blobs (chapters, audio, browse snapshots) in MinIO. SvelteKit frontend.

Architecture

scraper/
├── cmd/scraper/main.go               # Entry point: run | refresh | serve | save-browse
├── internal/
│   ├── orchestrator/orchestrator.go  # Catalogue walk → per-book metadata goroutines → chapter worker pool
│   ├── browser/                      # BrowserClient interface + direct HTTP (production) + Browserless variants
│   ├── novelfire/scraper.go          # novelfire.net scraping (catalogue, metadata, chapters, ranking)
│   ├── server/                       # HTTP API server (server.go + 6 handler files)
│   │   ├── server.go                 # Server struct, route registration, ListenAndServe
│   │   ├── handlers_scrape.go        # POST /scrape, /scrape/book, /scrape/book/range; job status/tasks
│   │   ├── handlers_browse.go        # GET /api/browse, /api/search, /api/cover — MinIO-cached browse pages
│   │   ├── handlers_preview.go       # GET /api/book-preview, /api/chapter-text-preview — live scrape, no store writes
│   │   ├── handlers_audio.go         # POST /api/audio, GET /api/audio-proxy, voice samples, presign
│   │   ├── handlers_progress.go      # GET/POST/DELETE /api/progress
│   │   ├── handlers_ranking.go       # GET /api/ranking, /api/cover
│   │   └── helpers.go                # stripMarkdown, hardcoded voice list fallback
│   ├── storage/                      # Persistence layer (PocketBase + MinIO)
│   │   ├── store.go                  # Store interface — single abstraction for server + orchestrator
│   │   ├── hybrid.go                 # HybridStore: routes structured data → PocketBase, blobs → MinIO
│   │   ├── pocketbase.go             # PocketBase REST admin client (7 collections, auth, schema bootstrap)
│   │   ├── minio.go                  # MinIO client (3 buckets: chapters, audio, browse)
│   │   └── coverutil.go              # Best-effort cover image downloader → browse bucket
│   └── scraper/
│       ├── interfaces.go             # NovelScraper interface + domain types (BookMeta, ChapterRef, etc.)
│       └── htmlutil/htmlutil.go      # HTML parsing helpers (NodeToMarkdown, ResolveURL, etc.)

Key Concepts

• Orchestrator: Catalogue stream → per-book goroutines (metadata + chapter list) → shared chapter work channel → N worker goroutines (chapter text). Scrape jobs tracked in PocketBase scraping_tasks.
• Storage: HybridStore implements the Store interface. PocketBase holds structured records (books, chapters_idx, ranking, progress, audio_cache, app_users, scraping_tasks). MinIO holds blobs (chapter markdown, audio MP3s, browse HTML snapshots, cover images).
• Browser Client: Production uses NewDirectHTTPClient (plain HTTP, no Browserless). Browserless variants (content/scrape/cdp) exist in browser/ but are only wired for the save-browse subcommand.
• Preview: GET /api/book-preview/{slug} scrapes metadata + chapter list live without persisting anything — used when a book is not yet in the library. On first visit, metadata and chapter index are auto-saved to PocketBase in the background.
• Server: 24 HTTP endpoints. Async scrape jobs (mutex, 409 on concurrent), in-flight dedup for audio generation, MinIO-backed browse page cache with mem-cache fallback.

Commands

# Build
cd scraper && go build -o bin/scraper ./cmd/scraper

# Full catalogue scrape (one-shot)
./bin/scraper run

# Single book
./bin/scraper run --url https://novelfire.net/book/xxx

# Re-scrape a book already in the DB (uses stored source_url)
./bin/scraper refresh <slug>

# HTTP server
./bin/scraper serve

# Capture browse pages to MinIO via SingleFile CLI (requires SINGLEFILE_PATH + BROWSERLESS_URL)
./bin/scraper save-browse

# Tests (unit only — integration tests require live services)
cd scraper && go test ./... -short

# All tests (requires MinIO + PocketBase + Browserless)
cd scraper && go test ./...

Environment Variables

Scraper (Go)

Variable	Description	Default
LOG_LEVEL	debug|info|warn|error	info
SCRAPER_HTTP_ADDR	HTTP listen address	:8080
SCRAPER_WORKERS	Chapter goroutines	NumCPU
SCRAPER_TIMEOUT	Per-request HTTP timeout (seconds)	90
KOKORO_URL	Kokoro-FastAPI TTS base URL	https://kokoro.kalekber.cc
KOKORO_VOICE	Default TTS voice	af_bella
MINIO_ENDPOINT	MinIO S3 API host:port	localhost:9000
MINIO_PUBLIC_ENDPOINT	Public MinIO endpoint for presigned URLs	""
MINIO_ACCESS_KEY	MinIO access key	admin
MINIO_SECRET_KEY	MinIO secret key	changeme123
MINIO_USE_SSL	TLS for internal MinIO connection	false
MINIO_PUBLIC_USE_SSL	TLS for public presigned URL endpoint	true
MINIO_BUCKET_CHAPTERS	Chapter markdown bucket	libnovel-chapters
MINIO_BUCKET_AUDIO	Audio MP3 bucket	libnovel-audio
MINIO_BUCKET_BROWSE	Browse HTML + cover image bucket	libnovel-browse
POCKETBASE_URL	PocketBase base URL	http://localhost:8090
POCKETBASE_ADMIN_EMAIL	PocketBase admin email	admin@libnovel.local
POCKETBASE_ADMIN_PASSWORD	PocketBase admin password	changeme123
BROWSERLESS_URL	Browserless WS endpoint (save-browse only)	http://localhost:3030
SINGLEFILE_PATH	SingleFile CLI binary path (save-browse only)	single-file

UI (SvelteKit)

Variable	Description	Default
AUTH_SECRET	HMAC signing secret for auth tokens	dev_secret_change_in_production
SCRAPER_API_URL	Internal URL of the Go scraper	http://localhost:8080
POCKETBASE_URL	PocketBase base URL	http://localhost:8090
POCKETBASE_ADMIN_EMAIL	PocketBase admin email	admin@libnovel.local
POCKETBASE_ADMIN_PASSWORD	PocketBase admin password	changeme123
PUBLIC_MINIO_PUBLIC_URL	Browser-visible MinIO URL (presigned links)	http://localhost:9000

Docker

docker-compose up -d  # Starts: minio, minio-init, pocketbase, pb-init, scraper, ui

Services:

Service	Port(s)	Role
minio	9000 (S3 API), 9001 (console)	Object storage
minio-init	—	One-shot bucket creation then exits
pocketbase	8090	Structured data store
pb-init	—	One-shot PocketBase collection bootstrap then exits
scraper	8080	Go scraper HTTP API
ui	5252 → internal 3000	SvelteKit frontend

Kokoro and Browserless are external services — not in docker-compose.

HTTP API Endpoints (Go scraper)

Method	Path	Description
GET	/health	Liveness probe
POST	/scrape	Enqueue full catalogue scrape
POST	/scrape/book	Enqueue single-book scrape {url}
POST	/scrape/book/range	Enqueue range scrape {url, from, to?}
GET	/api/scrape/status	Current scrape job status
GET	/api/scrape/tasks	All scrape task records
GET	/api/browse	Browse novelfire catalogue (MinIO-cached)
GET	/api/search	Search local + remote ?q=
GET	/api/ranking	Ranking list
GET	/api/cover/{domain}/{slug}	Proxy cover image from MinIO
GET	/api/book-preview/{slug}	Live metadata + chapter list (no store write)
GET	/api/chapter-text-preview/{slug}/{n}	Live chapter text (no store write)
POST	/api/reindex/{slug}	Rebuild chapters_idx from MinIO
GET	/api/chapter-text/{slug}/{n}	Chapter text (markdown stripped)
POST	/api/audio/{slug}/{n}	Trigger Kokoro TTS generation
GET	/api/audio-proxy/{slug}/{n}	Proxy generated audio
POST	/api/audio/voice-samples	Pre-generate voice samples
GET	/api/voices	List available Kokoro voices
GET	/api/presign/chapter/{slug}/{n}	Presigned MinIO URL for chapter
GET	/api/presign/audio/{slug}/{n}	Presigned MinIO URL for audio
GET	/api/presign/voice-sample/{voice}	Presigned MinIO URL for voice sample
GET	/api/progress	Get reading progress (session-scoped)
POST	/api/progress/{slug}	Set reading progress
DELETE	/api/progress/{slug}	Delete reading progress

Code Patterns

• log/slog for structured logging throughout
• Context-based cancellation on all network calls and goroutines
• Worker pool pattern in orchestrator (buffered channel + WaitGroup)
• Single async scrape job enforced by mutex; 409 on concurrent requests; job state persisted to scraping_tasks in PocketBase
• Store interface decouples all persistence — pass it around, never touch MinIO/PocketBase clients directly outside storage/
• Auth: custom HMAC-signed token (userId:username:role.<sig>) in libnovel_auth cookie; signed with AUTH_SECRET

AI Context Tips

• Primary files to modify: orchestrator.go, server/handlers_*.go, novelfire/scraper.go, storage/hybrid.go, storage/pocketbase.go
• To add a new scrape source: implement NovelScraper from internal/scraper/interfaces.go
• To add a new API endpoint: add handler in the appropriate handlers_*.go file, register in server.go ListenAndServe()
• Storage changes: update Store interface in store.go, implement on HybridStore (hybrid.go) and PocketBaseStore/MinioClient as needed; update mock in orchestrator_test.go
• Skip: scraper/bin/ (compiled binary), MinIO/PocketBase data volumes