diff --git a/AGENTS.md b/AGENTS.md index 23e4bae..0ef6d7d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,28 +1,43 @@ # libnovel Project -Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI. +Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI. Structured data in PocketBase, binary blobs (chapters, audio, browse snapshots) in MinIO. SvelteKit frontend. ## Architecture ``` scraper/ -├── cmd/scraper/main.go # Entry point: 'run' (one-shot) and 'serve' (HTTP server) +├── cmd/scraper/main.go # Entry point: run | refresh | serve | save-browse ├── internal/ -│ ├── orchestrator/orchestrator.go # Coordinates catalogue walk, metadata extraction, chapter scraping -│ ├── browser/ # Browser client (content/scrape/cdp strategies) via Browserless -│ ├── novelfire/scraper.go # novelfire.net specific scraping logic -│ ├── server/server.go # HTTP API (POST /scrape, POST /scrape/book) -│ ├── writer/writer.go # File writer (metadata.yaml, chapter .md files) -│ └── scraper/interfaces.go # NovelScraper interface definition -└── static/books/ # Output directory for scraped content +│ ├── orchestrator/orchestrator.go # Catalogue walk → per-book metadata goroutines → chapter worker pool +│ ├── browser/ # BrowserClient interface + direct HTTP (production) + Browserless variants +│ ├── novelfire/scraper.go # novelfire.net scraping (catalogue, metadata, chapters, ranking) +│ ├── server/ # HTTP API server (server.go + 6 handler files) +│ │ ├── server.go # Server struct, route registration, ListenAndServe +│ │ ├── handlers_scrape.go # POST /scrape, /scrape/book, /scrape/book/range; job status/tasks +│ │ ├── handlers_browse.go # GET /api/browse, /api/search, /api/cover — MinIO-cached browse pages +│ │ ├── handlers_preview.go # GET /api/book-preview, /api/chapter-text-preview — live scrape, no store writes +│ │ ├── handlers_audio.go # POST /api/audio, GET /api/audio-proxy, voice samples, presign +│ │ ├── handlers_progress.go # GET/POST/DELETE /api/progress +│ │ ├── handlers_ranking.go # GET /api/ranking, /api/cover +│ │ └── helpers.go # stripMarkdown, hardcoded voice list fallback +│ ├── storage/ # Persistence layer (PocketBase + MinIO) +│ │ ├── store.go # Store interface — single abstraction for server + orchestrator +│ │ ├── hybrid.go # HybridStore: routes structured data → PocketBase, blobs → MinIO +│ │ ├── pocketbase.go # PocketBase REST admin client (7 collections, auth, schema bootstrap) +│ │ ├── minio.go # MinIO client (3 buckets: chapters, audio, browse) +│ │ └── coverutil.go # Best-effort cover image downloader → browse bucket +│ └── scraper/ +│ ├── interfaces.go # NovelScraper interface + domain types (BookMeta, ChapterRef, etc.) +│ └── htmlutil/htmlutil.go # HTML parsing helpers (NodeToMarkdown, ResolveURL, etc.) ``` ## Key Concepts -- **Orchestrator**: Manages concurrency - catalogue streaming → per-book metadata goroutines → chapter worker pool -- **Browser Client**: 3 strategies (content/scrape/cdp) via Browserless Chrome container -- **Writer**: Writes metadata.yaml and chapter markdown files to `static/books/{slug}/vol-0/1-50/` -- **Server**: HTTP API with async scrape jobs, UI for browsing books/chapters, chapter-text endpoint for TTS +- **Orchestrator**: Catalogue stream → per-book goroutines (metadata + chapter list) → shared chapter work channel → N worker goroutines (chapter text). Scrape jobs tracked in PocketBase `scraping_tasks`. +- **Storage**: `HybridStore` implements the `Store` interface. PocketBase holds structured records (`books`, `chapters_idx`, `ranking`, `progress`, `audio_cache`, `app_users`, `scraping_tasks`). MinIO holds blobs (chapter markdown, audio MP3s, browse HTML snapshots, cover images). +- **Browser Client**: Production uses `NewDirectHTTPClient` (plain HTTP, no Browserless). Browserless variants (content/scrape/cdp) exist in `browser/` but are only wired for the `save-browse` subcommand. +- **Preview**: `GET /api/book-preview/{slug}` scrapes metadata + chapter list live without persisting anything — used when a book is not yet in the library. On first visit, metadata and chapter index are auto-saved to PocketBase in the background. +- **Server**: 24 HTTP endpoints. Async scrape jobs (mutex, 409 on concurrent), in-flight dedup for audio generation, MinIO-backed browse page cache with mem-cache fallback. ## Commands @@ -30,60 +45,127 @@ scraper/ # Build cd scraper && go build -o bin/scraper ./cmd/scraper -# One-shot scrape (full catalogue) +# Full catalogue scrape (one-shot) ./bin/scraper run # Single book ./bin/scraper run --url https://novelfire.net/book/xxx +# Re-scrape a book already in the DB (uses stored source_url) +./bin/scraper refresh + # HTTP server ./bin/scraper serve -# Tests +# Capture browse pages to MinIO via SingleFile CLI (requires SINGLEFILE_PATH + BROWSERLESS_URL) +./bin/scraper save-browse + +# Tests (unit only — integration tests require live services) +cd scraper && go test ./... -short + +# All tests (requires MinIO + PocketBase + Browserless) cd scraper && go test ./... ``` ## Environment Variables +### Scraper (Go) + | Variable | Description | Default | |----------|-------------|---------| -| BROWSERLESS_URL | Browserless Chrome endpoint | http://localhost:3030 | -| BROWSERLESS_STRATEGY | content \| scrape \| cdp | content | -| SCRAPER_WORKERS | Chapter goroutines | NumCPU | -| SCRAPER_STATIC_ROOT | Output directory | ./static/books | -| SCRAPER_HTTP_ADDR | HTTP listen address | :8080 | -| KOKORO_URL | Kokoro TTS endpoint | http://localhost:8880 | -| KOKORO_VOICE | Default TTS voice | af_bella | -| LOG_LEVEL | debug \| info \| warn \| error | info | +| `LOG_LEVEL` | `debug\|info\|warn\|error` | `info` | +| `SCRAPER_HTTP_ADDR` | HTTP listen address | `:8080` | +| `SCRAPER_WORKERS` | Chapter goroutines | `NumCPU` | +| `SCRAPER_TIMEOUT` | Per-request HTTP timeout (seconds) | `90` | +| `KOKORO_URL` | Kokoro-FastAPI TTS base URL | `https://kokoro.kalekber.cc` | +| `KOKORO_VOICE` | Default TTS voice | `af_bella` | +| `MINIO_ENDPOINT` | MinIO S3 API host:port | `localhost:9000` | +| `MINIO_PUBLIC_ENDPOINT` | Public MinIO endpoint for presigned URLs | `""` | +| `MINIO_ACCESS_KEY` | MinIO access key | `admin` | +| `MINIO_SECRET_KEY` | MinIO secret key | `changeme123` | +| `MINIO_USE_SSL` | TLS for internal MinIO connection | `false` | +| `MINIO_PUBLIC_USE_SSL` | TLS for public presigned URL endpoint | `true` | +| `MINIO_BUCKET_CHAPTERS` | Chapter markdown bucket | `libnovel-chapters` | +| `MINIO_BUCKET_AUDIO` | Audio MP3 bucket | `libnovel-audio` | +| `MINIO_BUCKET_BROWSE` | Browse HTML + cover image bucket | `libnovel-browse` | +| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` | +| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` | +| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` | +| `BROWSERLESS_URL` | Browserless WS endpoint (save-browse only) | `http://localhost:3030` | +| `SINGLEFILE_PATH` | SingleFile CLI binary path (save-browse only) | `single-file` | + +### UI (SvelteKit) + +| Variable | Description | Default | +|----------|-------------|---------| +| `AUTH_SECRET` | HMAC signing secret for auth tokens | `dev_secret_change_in_production` | +| `SCRAPER_API_URL` | Internal URL of the Go scraper | `http://localhost:8080` | +| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` | +| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` | +| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` | +| `PUBLIC_MINIO_PUBLIC_URL` | Browser-visible MinIO URL (presigned links) | `http://localhost:9000` | ## Docker ```bash -docker-compose up -d # Starts browserless, kokoro, scraper +docker-compose up -d # Starts: minio, minio-init, pocketbase, pb-init, scraper, ui ``` +Services: + +| Service | Port(s) | Role | +|---------|---------|------| +| `minio` | `9000` (S3 API), `9001` (console) | Object storage | +| `minio-init` | — | One-shot bucket creation then exits | +| `pocketbase` | `8090` | Structured data store | +| `pb-init` | — | One-shot PocketBase collection bootstrap then exits | +| `scraper` | `8080` | Go scraper HTTP API | +| `ui` | `5252` → internal `3000` | SvelteKit frontend | + +Kokoro and Browserless are **external services** — not in docker-compose. + +## HTTP API Endpoints (Go scraper) + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/health` | Liveness probe | +| `POST` | `/scrape` | Enqueue full catalogue scrape | +| `POST` | `/scrape/book` | Enqueue single-book scrape `{url}` | +| `POST` | `/scrape/book/range` | Enqueue range scrape `{url, from, to?}` | +| `GET` | `/api/scrape/status` | Current scrape job status | +| `GET` | `/api/scrape/tasks` | All scrape task records | +| `GET` | `/api/browse` | Browse novelfire catalogue (MinIO-cached) | +| `GET` | `/api/search` | Search local + remote `?q=` | +| `GET` | `/api/ranking` | Ranking list | +| `GET` | `/api/cover/{domain}/{slug}` | Proxy cover image from MinIO | +| `GET` | `/api/book-preview/{slug}` | Live metadata + chapter list (no store write) | +| `GET` | `/api/chapter-text-preview/{slug}/{n}` | Live chapter text (no store write) | +| `POST` | `/api/reindex/{slug}` | Rebuild chapters_idx from MinIO | +| `GET` | `/api/chapter-text/{slug}/{n}` | Chapter text (markdown stripped) | +| `POST` | `/api/audio/{slug}/{n}` | Trigger Kokoro TTS generation | +| `GET` | `/api/audio-proxy/{slug}/{n}` | Proxy generated audio | +| `POST` | `/api/audio/voice-samples` | Pre-generate voice samples | +| `GET` | `/api/voices` | List available Kokoro voices | +| `GET` | `/api/presign/chapter/{slug}/{n}` | Presigned MinIO URL for chapter | +| `GET` | `/api/presign/audio/{slug}/{n}` | Presigned MinIO URL for audio | +| `GET` | `/api/presign/voice-sample/{voice}` | Presigned MinIO URL for voice sample | +| `GET` | `/api/progress` | Get reading progress (session-scoped) | +| `POST` | `/api/progress/{slug}` | Set reading progress | +| `DELETE` | `/api/progress/{slug}` | Delete reading progress | + ## Code Patterns -- Uses `log/slog` for structured logging -- Context-based cancellation throughout -- Worker pool pattern in orchestrator (channel + goroutines) -- Mutex for single async job (409 on concurrent scrape requests) +- `log/slog` for structured logging throughout +- Context-based cancellation on all network calls and goroutines +- Worker pool pattern in orchestrator (buffered channel + WaitGroup) +- Single async scrape job enforced by mutex; 409 on concurrent requests; job state persisted to `scraping_tasks` in PocketBase +- `Store` interface decouples all persistence — pass it around, never touch MinIO/PocketBase clients directly outside `storage/` +- Auth: custom HMAC-signed token (`userId:username:role.`) in `libnovel_auth` cookie; signed with `AUTH_SECRET` ## AI Context Tips -- Primary files to modify: `orchestrator.go`, `server.go`, `scraper.go`, `browser/*.go` -- To add new source: implement `NovelScraper` interface from `internal/scraper/interfaces.go` -- Skip `static/` directory - generated content, not source - -## Speed Up AI Sessions (Optional) - -For faster AI context loading, use **Context7** (free, local indexing): - -```bash -# Install and index once -npx @context7/cli@latest index --path . --ignore .aiignore - -# After first run, AI tools will query the index instead of re-scanning files -``` - -VSCode extension: https://marketplace.visualstudio.com/items?itemName=context7.context7 +- **Primary files to modify**: `orchestrator.go`, `server/handlers_*.go`, `novelfire/scraper.go`, `storage/hybrid.go`, `storage/pocketbase.go` +- **To add a new scrape source**: implement `NovelScraper` from `internal/scraper/interfaces.go` +- **To add a new API endpoint**: add handler in the appropriate `handlers_*.go` file, register in `server.go` `ListenAndServe()` +- **Storage changes**: update `Store` interface in `store.go`, implement on `HybridStore` (hybrid.go) and `PocketBaseStore`/`MinioClient` as needed; update mock in `orchestrator_test.go` +- **Skip**: `scraper/bin/` (compiled binary), MinIO/PocketBase data volumes