docs: update AGENTS.md to reflect current architecture

AGENTS.md

@@ -1,28 +1,43 @@
 # libnovel Project
 
-Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI.
+Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI. Structured data in PocketBase, binary blobs (chapters, audio, browse snapshots) in MinIO. SvelteKit frontend.
 
 ## Architecture
 
 ```
 scraper/
-├── cmd/scraper/main.go           # Entry point: 'run' (one-shot) and 'serve' (HTTP server)
+├── cmd/scraper/main.go               # Entry point: run | refresh | serve | save-browse
 ├── internal/
-│   ├── orchestrator/orchestrator.go  # Coordinates catalogue walk, metadata extraction, chapter scraping
+│   ├── orchestrator/orchestrator.go  # Catalogue walk → per-book metadata goroutines → chapter worker pool
-│   ├── browser/                       # Browser client (content/scrape/cdp strategies) via Browserless
+│   ├── browser/                      # BrowserClient interface + direct HTTP (production) + Browserless variants
-│   ├── novelfire/scraper.go          # novelfire.net specific scraping logic
+│   ├── novelfire/scraper.go          # novelfire.net scraping (catalogue, metadata, chapters, ranking)
-│   ├── server/server.go              # HTTP API (POST /scrape, POST /scrape/book)
+│   ├── server/                       # HTTP API server (server.go + 6 handler files)
-│   ├── writer/writer.go              # File writer (metadata.yaml, chapter .md files)
+│   │   ├── server.go                 # Server struct, route registration, ListenAndServe
-│   └── scraper/interfaces.go         # NovelScraper interface definition
+│   │   ├── handlers_scrape.go        # POST /scrape, /scrape/book, /scrape/book/range; job status/tasks
-└── static/books/                     # Output directory for scraped content
+│   │   ├── handlers_browse.go        # GET /api/browse, /api/search, /api/cover — MinIO-cached browse pages
+│   │   ├── handlers_preview.go       # GET /api/book-preview, /api/chapter-text-preview — live scrape, no store writes
+│   │   ├── handlers_audio.go         # POST /api/audio, GET /api/audio-proxy, voice samples, presign
+│   │   ├── handlers_progress.go      # GET/POST/DELETE /api/progress
+│   │   ├── handlers_ranking.go       # GET /api/ranking, /api/cover
+│   │   └── helpers.go                # stripMarkdown, hardcoded voice list fallback
+│   ├── storage/                      # Persistence layer (PocketBase + MinIO)
+│   │   ├── store.go                  # Store interface — single abstraction for server + orchestrator
+│   │   ├── hybrid.go                 # HybridStore: routes structured data → PocketBase, blobs → MinIO
+│   │   ├── pocketbase.go             # PocketBase REST admin client (7 collections, auth, schema bootstrap)
+│   │   ├── minio.go                  # MinIO client (3 buckets: chapters, audio, browse)
+│   │   └── coverutil.go              # Best-effort cover image downloader → browse bucket
+│   └── scraper/
+│       ├── interfaces.go             # NovelScraper interface + domain types (BookMeta, ChapterRef, etc.)
+│       └── htmlutil/htmlutil.go      # HTML parsing helpers (NodeToMarkdown, ResolveURL, etc.)
 ```
 
 ## Key Concepts
 
-- **Orchestrator**: Manages concurrency - catalogue streaming → per-book metadata goroutines → chapter worker pool
+- **Orchestrator**: Catalogue stream → per-book goroutines (metadata + chapter list) → shared chapter work channel → N worker goroutines (chapter text). Scrape jobs tracked in PocketBase `scraping_tasks`.
-- **Browser Client**: 3 strategies (content/scrape/cdp) via Browserless Chrome container
+- **Storage**: `HybridStore` implements the `Store` interface. PocketBase holds structured records (`books`, `chapters_idx`, `ranking`, `progress`, `audio_cache`, `app_users`, `scraping_tasks`). MinIO holds blobs (chapter markdown, audio MP3s, browse HTML snapshots, cover images).
-- **Writer**: Writes metadata.yaml and chapter markdown files to `static/books/{slug}/vol-0/1-50/`
+- **Browser Client**: Production uses `NewDirectHTTPClient` (plain HTTP, no Browserless). Browserless variants (content/scrape/cdp) exist in `browser/` but are only wired for the `save-browse` subcommand.
-- **Server**: HTTP API with async scrape jobs, UI for browsing books/chapters, chapter-text endpoint for TTS
+- **Preview**: `GET /api/book-preview/{slug}` scrapes metadata + chapter list live without persisting anything — used when a book is not yet in the library. On first visit, metadata and chapter index are auto-saved to PocketBase in the background.
+- **Server**: 24 HTTP endpoints. Async scrape jobs (mutex, 409 on concurrent), in-flight dedup for audio generation, MinIO-backed browse page cache with mem-cache fallback.
 
 ## Commands
 
@@ -30,60 +45,127 @@ scraper/
 # Build
 cd scraper && go build -o bin/scraper ./cmd/scraper
 
-# One-shot scrape (full catalogue)
+# Full catalogue scrape (one-shot)
 ./bin/scraper run
 
 # Single book
 ./bin/scraper run --url https://novelfire.net/book/xxx
 
+# Re-scrape a book already in the DB (uses stored source_url)
+./bin/scraper refresh <slug>
+
 # HTTP server
 ./bin/scraper serve
 
-# Tests
+# Capture browse pages to MinIO via SingleFile CLI (requires SINGLEFILE_PATH + BROWSERLESS_URL)
+./bin/scraper save-browse
+
+# Tests (unit only — integration tests require live services)
+cd scraper && go test ./... -short
+
+# All tests (requires MinIO + PocketBase + Browserless)
 cd scraper && go test ./...
 ```
 
 ## Environment Variables
 
+### Scraper (Go)
+
 | Variable | Description | Default |
 |----------|-------------|---------|
-| BROWSERLESS_URL | Browserless Chrome endpoint | http://localhost:3030 |
+| `LOG_LEVEL` | `debug\|info\|warn\|error` | `info` |
-| BROWSERLESS_STRATEGY | content \| scrape \| cdp | content |
+| `SCRAPER_HTTP_ADDR` | HTTP listen address | `:8080` |
-| SCRAPER_WORKERS | Chapter goroutines | NumCPU |
+| `SCRAPER_WORKERS` | Chapter goroutines | `NumCPU` |
-| SCRAPER_STATIC_ROOT | Output directory | ./static/books |
+| `SCRAPER_TIMEOUT` | Per-request HTTP timeout (seconds) | `90` |
-| SCRAPER_HTTP_ADDR | HTTP listen address | :8080 |
+| `KOKORO_URL` | Kokoro-FastAPI TTS base URL | `https://kokoro.kalekber.cc` |
-| KOKORO_URL | Kokoro TTS endpoint | http://localhost:8880 |
+| `KOKORO_VOICE` | Default TTS voice | `af_bella` |
-| KOKORO_VOICE | Default TTS voice | af_bella |
+| `MINIO_ENDPOINT` | MinIO S3 API host:port | `localhost:9000` |
-| LOG_LEVEL | debug \| info \| warn \| error | info |
+| `MINIO_PUBLIC_ENDPOINT` | Public MinIO endpoint for presigned URLs | `""` |
+| `MINIO_ACCESS_KEY` | MinIO access key | `admin` |
+| `MINIO_SECRET_KEY` | MinIO secret key | `changeme123` |
+| `MINIO_USE_SSL` | TLS for internal MinIO connection | `false` |
+| `MINIO_PUBLIC_USE_SSL` | TLS for public presigned URL endpoint | `true` |
+| `MINIO_BUCKET_CHAPTERS` | Chapter markdown bucket | `libnovel-chapters` |
+| `MINIO_BUCKET_AUDIO` | Audio MP3 bucket | `libnovel-audio` |
+| `MINIO_BUCKET_BROWSE` | Browse HTML + cover image bucket | `libnovel-browse` |
+| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` |
+| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` |
+| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` |
+| `BROWSERLESS_URL` | Browserless WS endpoint (save-browse only) | `http://localhost:3030` |
+| `SINGLEFILE_PATH` | SingleFile CLI binary path (save-browse only) | `single-file` |
+
+### UI (SvelteKit)
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `AUTH_SECRET` | HMAC signing secret for auth tokens | `dev_secret_change_in_production` |
+| `SCRAPER_API_URL` | Internal URL of the Go scraper | `http://localhost:8080` |
+| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` |
+| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` |
+| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` |
+| `PUBLIC_MINIO_PUBLIC_URL` | Browser-visible MinIO URL (presigned links) | `http://localhost:9000` |
 
 ## Docker
 
 ```bash
-docker-compose up -d  # Starts browserless, kokoro, scraper
+docker-compose up -d  # Starts: minio, minio-init, pocketbase, pb-init, scraper, ui
 ```
 
+Services:
+
+| Service | Port(s) | Role |
+|---------|---------|------|
+| `minio` | `9000` (S3 API), `9001` (console) | Object storage |
+| `minio-init` | — | One-shot bucket creation then exits |
+| `pocketbase` | `8090` | Structured data store |
+| `pb-init` | — | One-shot PocketBase collection bootstrap then exits |
+| `scraper` | `8080` | Go scraper HTTP API |
+| `ui` | `5252` → internal `3000` | SvelteKit frontend |
+
+Kokoro and Browserless are **external services** — not in docker-compose.
+
+## HTTP API Endpoints (Go scraper)
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/health` | Liveness probe |
+| `POST` | `/scrape` | Enqueue full catalogue scrape |
+| `POST` | `/scrape/book` | Enqueue single-book scrape `{url}` |
+| `POST` | `/scrape/book/range` | Enqueue range scrape `{url, from, to?}` |
+| `GET` | `/api/scrape/status` | Current scrape job status |
+| `GET` | `/api/scrape/tasks` | All scrape task records |
+| `GET` | `/api/browse` | Browse novelfire catalogue (MinIO-cached) |
+| `GET` | `/api/search` | Search local + remote `?q=` |
+| `GET` | `/api/ranking` | Ranking list |
+| `GET` | `/api/cover/{domain}/{slug}` | Proxy cover image from MinIO |
+| `GET` | `/api/book-preview/{slug}` | Live metadata + chapter list (no store write) |
+| `GET` | `/api/chapter-text-preview/{slug}/{n}` | Live chapter text (no store write) |
+| `POST` | `/api/reindex/{slug}` | Rebuild chapters_idx from MinIO |
+| `GET` | `/api/chapter-text/{slug}/{n}` | Chapter text (markdown stripped) |
+| `POST` | `/api/audio/{slug}/{n}` | Trigger Kokoro TTS generation |
+| `GET` | `/api/audio-proxy/{slug}/{n}` | Proxy generated audio |
+| `POST` | `/api/audio/voice-samples` | Pre-generate voice samples |
+| `GET` | `/api/voices` | List available Kokoro voices |
+| `GET` | `/api/presign/chapter/{slug}/{n}` | Presigned MinIO URL for chapter |
+| `GET` | `/api/presign/audio/{slug}/{n}` | Presigned MinIO URL for audio |
+| `GET` | `/api/presign/voice-sample/{voice}` | Presigned MinIO URL for voice sample |
+| `GET` | `/api/progress` | Get reading progress (session-scoped) |
+| `POST` | `/api/progress/{slug}` | Set reading progress |
+| `DELETE` | `/api/progress/{slug}` | Delete reading progress |
+
 ## Code Patterns
 
-- Uses `log/slog` for structured logging
+- `log/slog` for structured logging throughout
-- Context-based cancellation throughout
+- Context-based cancellation on all network calls and goroutines
-- Worker pool pattern in orchestrator (channel + goroutines)
+- Worker pool pattern in orchestrator (buffered channel + WaitGroup)
-- Mutex for single async job (409 on concurrent scrape requests)
+- Single async scrape job enforced by mutex; 409 on concurrent requests; job state persisted to `scraping_tasks` in PocketBase
+- `Store` interface decouples all persistence — pass it around, never touch MinIO/PocketBase clients directly outside `storage/`
+- Auth: custom HMAC-signed token (`userId:username:role.<sig>`) in `libnovel_auth` cookie; signed with `AUTH_SECRET`
 
 ## AI Context Tips
 
-- Primary files to modify: `orchestrator.go`, `server.go`, `scraper.go`, `browser/*.go`
+- **Primary files to modify**: `orchestrator.go`, `server/handlers_*.go`, `novelfire/scraper.go`, `storage/hybrid.go`, `storage/pocketbase.go`
-- To add new source: implement `NovelScraper` interface from `internal/scraper/interfaces.go`
+- **To add a new scrape source**: implement `NovelScraper` from `internal/scraper/interfaces.go`
-- Skip `static/` directory - generated content, not source
+- **To add a new API endpoint**: add handler in the appropriate `handlers_*.go` file, register in `server.go` `ListenAndServe()`
-
+- **Storage changes**: update `Store` interface in `store.go`, implement on `HybridStore` (hybrid.go) and `PocketBaseStore`/`MinioClient` as needed; update mock in `orchestrator_test.go`
-## Speed Up AI Sessions (Optional)
+- **Skip**: `scraper/bin/` (compiled binary), MinIO/PocketBase data volumes
-
-For faster AI context loading, use **Context7** (free, local indexing):
-
-```bash
-# Install and index once
-npx @context7/cli@latest index --path . --ignore .aiignore
-
-# After first run, AI tools will query the index instead of re-scanning files
-```
-
-VSCode extension: https://marketplace.visualstudio.com/items?itemName=context7.context7