# libnovel Project

Go web scraper for novelfire.net with TTS support via Kokoro-FastAPI. Structured data in PocketBase, binary blobs (chapters, audio, browse snapshots) in MinIO. SvelteKit frontend.

## Architecture

```
scraper/
├── cmd/scraper/main.go               # Entry point: run | refresh | serve | save-browse
├── internal/
│   ├── orchestrator/orchestrator.go  # Catalogue walk → per-book metadata goroutines → chapter worker pool
│   ├── browser/                      # BrowserClient interface + direct HTTP (production) + Browserless variants
│   ├── novelfire/scraper.go          # novelfire.net scraping (catalogue, metadata, chapters, ranking)
│   ├── server/                       # HTTP API server (server.go + 6 handler files)
│   │   ├── server.go                 # Server struct, route registration, ListenAndServe
│   │   ├── handlers_scrape.go        # POST /scrape, /scrape/book, /scrape/book/range; job status/tasks
│   │   ├── handlers_browse.go        # GET /api/browse, /api/search, /api/cover — MinIO-cached browse pages
│   │   ├── handlers_preview.go       # GET /api/book-preview, /api/chapter-text-preview — live scrape, no store writes
│   │   ├── handlers_audio.go         # POST /api/audio, GET /api/audio-proxy, voice samples, presign
│   │   ├── handlers_progress.go      # GET/POST/DELETE /api/progress
│   │   ├── handlers_ranking.go       # GET /api/ranking, /api/cover
│   │   └── helpers.go                # stripMarkdown, hardcoded voice list fallback
│   ├── storage/                      # Persistence layer (PocketBase + MinIO)
│   │   ├── store.go                  # Store interface — single abstraction for server + orchestrator
│   │   ├── hybrid.go                 # HybridStore: routes structured data → PocketBase, blobs → MinIO
│   │   ├── pocketbase.go             # PocketBase REST admin client (7 collections, auth, schema bootstrap)
│   │   ├── minio.go                  # MinIO client (3 buckets: chapters, audio, browse)
│   │   └── coverutil.go              # Best-effort cover image downloader → browse bucket
│   └── scraper/
│       ├── interfaces.go             # NovelScraper interface + domain types (BookMeta, ChapterRef, etc.)
│       └── htmlutil/htmlutil.go      # HTML parsing helpers (NodeToMarkdown, ResolveURL, etc.)
```

## Key Concepts

- **Orchestrator**: Catalogue stream → per-book goroutines (metadata + chapter list) → shared chapter work channel → N worker goroutines (chapter text). Scrape jobs tracked in PocketBase `scraping_tasks`.
- **Storage**: `HybridStore` implements the `Store` interface. PocketBase holds structured records (`books`, `chapters_idx`, `ranking`, `progress`, `audio_cache`, `app_users`, `scraping_tasks`). MinIO holds blobs (chapter markdown, audio MP3s, browse HTML snapshots, cover images).
- **Browser Client**: Production uses `NewDirectHTTPClient` (plain HTTP, no Browserless). Browserless variants (content/scrape/cdp) exist in `browser/` but are only wired for the `save-browse` subcommand.
- **Preview**: `GET /api/book-preview/{slug}` scrapes metadata + chapter list live without persisting anything — used when a book is not yet in the library. On first visit, metadata and chapter index are auto-saved to PocketBase in the background.
- **Server**: 24 HTTP endpoints. Async scrape jobs (mutex, 409 on concurrent), in-flight dedup for audio generation, MinIO-backed browse page cache with mem-cache fallback.

## Commands

```bash
# Build
cd scraper && go build -o bin/scraper ./cmd/scraper

# Full catalogue scrape (one-shot)
./bin/scraper run

# Single book
./bin/scraper run --url https://novelfire.net/book/xxx

# Re-scrape a book already in the DB (uses stored source_url)
./bin/scraper refresh <slug>

# HTTP server
./bin/scraper serve

# Capture browse pages to MinIO via SingleFile CLI (requires SINGLEFILE_PATH + BROWSERLESS_URL)
./bin/scraper save-browse

# Tests (unit only — integration tests require live services)
cd scraper && go test ./... -short

# All tests (requires MinIO + PocketBase + Browserless)
cd scraper && go test ./...
```

## Environment Variables

### Scraper (Go)

| Variable | Description | Default |
|----------|-------------|---------|
| `LOG_LEVEL` | `debug\|info\|warn\|error` | `info` |
| `SCRAPER_HTTP_ADDR` | HTTP listen address | `:8080` |
| `SCRAPER_WORKERS` | Chapter goroutines | `NumCPU` |
| `SCRAPER_TIMEOUT` | Per-request HTTP timeout (seconds) | `90` |
| `KOKORO_URL` | Kokoro-FastAPI TTS base URL | `https://kokoro.kalekber.cc` |
| `KOKORO_VOICE` | Default TTS voice | `af_bella` |
| `MINIO_ENDPOINT` | MinIO S3 API host:port | `localhost:9000` |
| `MINIO_PUBLIC_ENDPOINT` | Public MinIO endpoint for presigned URLs | `""` |
| `MINIO_ACCESS_KEY` | MinIO access key | `admin` |
| `MINIO_SECRET_KEY` | MinIO secret key | `changeme123` |
| `MINIO_USE_SSL` | TLS for internal MinIO connection | `false` |
| `MINIO_PUBLIC_USE_SSL` | TLS for public presigned URL endpoint | `true` |
| `MINIO_BUCKET_CHAPTERS` | Chapter markdown bucket | `libnovel-chapters` |
| `MINIO_BUCKET_AUDIO` | Audio MP3 bucket | `libnovel-audio` |
| `MINIO_BUCKET_BROWSE` | Browse HTML + cover image bucket | `libnovel-browse` |
| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` |
| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` |
| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` |
| `BROWSERLESS_URL` | Browserless WS endpoint (save-browse only) | `http://localhost:3030` |
| `SINGLEFILE_PATH` | SingleFile CLI binary path (save-browse only) | `single-file` |

### UI (SvelteKit)

| Variable | Description | Default |
|----------|-------------|---------|
| `AUTH_SECRET` | HMAC signing secret for auth tokens | `dev_secret_change_in_production` |
| `SCRAPER_API_URL` | Internal URL of the Go scraper | `http://localhost:8080` |
| `POCKETBASE_URL` | PocketBase base URL | `http://localhost:8090` |
| `POCKETBASE_ADMIN_EMAIL` | PocketBase admin email | `admin@libnovel.local` |
| `POCKETBASE_ADMIN_PASSWORD` | PocketBase admin password | `changeme123` |
| `PUBLIC_MINIO_PUBLIC_URL` | Browser-visible MinIO URL (presigned links) | `http://localhost:9000` |

## Docker

```bash
docker-compose up -d  # Starts: minio, minio-init, pocketbase, pb-init, scraper, ui
```

Services:

| Service | Port(s) | Role |
|---------|---------|------|
| `minio` | `9000` (S3 API), `9001` (console) | Object storage |
| `minio-init` | — | One-shot bucket creation then exits |
| `pocketbase` | `8090` | Structured data store |
| `pb-init` | — | One-shot PocketBase collection bootstrap then exits |
| `scraper` | `8080` | Go scraper HTTP API |
| `ui` | `5252` → internal `3000` | SvelteKit frontend |

Kokoro and Browserless are **external services** — not in docker-compose.

## HTTP API Endpoints (Go scraper)

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Liveness probe |
| `POST` | `/scrape` | Enqueue full catalogue scrape |
| `POST` | `/scrape/book` | Enqueue single-book scrape `{url}` |
| `POST` | `/scrape/book/range` | Enqueue range scrape `{url, from, to?}` |
| `GET` | `/api/scrape/status` | Current scrape job status |
| `GET` | `/api/scrape/tasks` | All scrape task records |
| `GET` | `/api/browse` | Browse novelfire catalogue (MinIO-cached) |
| `GET` | `/api/search` | Search local + remote `?q=` |
| `GET` | `/api/ranking` | Ranking list |
| `GET` | `/api/cover/{domain}/{slug}` | Proxy cover image from MinIO |
| `GET` | `/api/book-preview/{slug}` | Live metadata + chapter list (no store write) |
| `GET` | `/api/chapter-text-preview/{slug}/{n}` | Live chapter text (no store write) |
| `POST` | `/api/reindex/{slug}` | Rebuild chapters_idx from MinIO |
| `GET` | `/api/chapter-text/{slug}/{n}` | Chapter text (markdown stripped) |
| `POST` | `/api/audio/{slug}/{n}` | Trigger Kokoro TTS generation |
| `GET` | `/api/audio-proxy/{slug}/{n}` | Proxy generated audio |
| `POST` | `/api/audio/voice-samples` | Pre-generate voice samples |
| `GET` | `/api/voices` | List available Kokoro voices |
| `GET` | `/api/presign/chapter/{slug}/{n}` | Presigned MinIO URL for chapter |
| `GET` | `/api/presign/audio/{slug}/{n}` | Presigned MinIO URL for audio |
| `GET` | `/api/presign/voice-sample/{voice}` | Presigned MinIO URL for voice sample |
| `GET` | `/api/progress` | Get reading progress (session-scoped) |
| `POST` | `/api/progress/{slug}` | Set reading progress |
| `DELETE` | `/api/progress/{slug}` | Delete reading progress |

## Code Patterns

- `log/slog` for structured logging throughout
- Context-based cancellation on all network calls and goroutines
- Worker pool pattern in orchestrator (buffered channel + WaitGroup)
- Single async scrape job enforced by mutex; 409 on concurrent requests; job state persisted to `scraping_tasks` in PocketBase
- `Store` interface decouples all persistence — pass it around, never touch MinIO/PocketBase clients directly outside `storage/`
- Auth: custom HMAC-signed token (`userId:username:role.<sig>`) in `libnovel_auth` cookie; signed with `AUTH_SECRET`

## AI Context Tips

- **Primary files to modify**: `orchestrator.go`, `server/handlers_*.go`, `novelfire/scraper.go`, `storage/hybrid.go`, `storage/pocketbase.go`
- **To add a new scrape source**: implement `NovelScraper` from `internal/scraper/interfaces.go`
- **To add a new API endpoint**: add handler in the appropriate `handlers_*.go` file, register in `server.go` `ListenAndServe()`
- **Storage changes**: update `Store` interface in `store.go`, implement on `HybridStore` (hybrid.go) and `PocketBaseStore`/`MinioClient` as needed; update mock in `orchestrator_test.go`
- **Skip**: `scraper/bin/` (compiled binary), MinIO/PocketBase data volumes

## iOS App

See `ios/AGENTS.md` for full iOS/SwiftUI conventions.

## Documentation Tools

This project has two MCP-backed documentation tools available. Use them proactively:

- **`context7`** — Live Apple SwiftUI/Swift docs, Go stdlib, SvelteKit, and any other library docs. Use before implementing anything non-trivial in Swift/SwiftUI. Example: `use context7 to look up NavigationStack`.
- **`gh_grep`** — Search real-world code on GitHub for implementation patterns. Example: `use gh_grep to find examples of background URLSession in Swift`.