libnovel/v3/TODO.md

# v3 Implementation Plan

v3 is a self-contained directory. All services live under `v3/`. Nothing shares
code with the root `backend/` or `ui-v2/` directories.

## New in v3 vs v2

| Addition | Why |
|----------|-----|
| **Caddy** reverse proxy | Single public entry point, automatic HTTPS via Let's Encrypt, no manual cert management |
| **Meilisearch** | Full-text search over locally scraped books; replaces live novelfire.net scrape on every search |
| **Valkey** (Redis-compatible) | Shared presign URL cache; replaces the per-process in-memory Map in the UI Node process |
| **Runner `/metrics` HTTP endpoint** | Exposes task counters (pending, running, failed, completed) for healthcheck / Caddy visibility |

## Directory layout (target)

```
v3/
├── backend/                   # copied from ../backend/, then modified
│   ├── cmd/
│   │   ├── backend/           # unchanged
│   │   └── runner/            # add metrics HTTP server
│   ├── internal/
│   │   ├── backend/
│   │   │   ├── server.go      # add /api/search endpoint (Meilisearch)
│   │   │   └── handlers.go    # add searchHandler, presign now reads Valkey
│   │   ├── runner/
│   │   │   ├── runner.go      # add atomic counters + metrics HTTP server
│   │   │   └── metrics.go     # NEW: MetricsServer (net/http, /metrics JSON)
│   │   ├── meili/             # NEW: Meilisearch client package
│   │   │   └── client.go
│   │   ├── presigncache/      # NEW: Valkey-backed presign cache
│   │   │   └── cache.go
│   │   └── config/
│   │       └── config.go      # add Meilisearch + Valkey + metrics addr vars
│   └── go.mod                 # add meilisearch-go, go-redis/v9
├── ui/                        # copied from ../ui-v2/, then modified
│   └── src/lib/server/
│       └── presignCache.ts    # replace in-process Map with Valkey (ioredis)
├── docs/
│   ├── architecture.d2        # DONE
│   └── architecture.mermaid.md # DONE
├── scripts/
│   └── pb-init-v3.sh          # same as v2 unless schema changes
├── Caddyfile                  # NEW
├── docker-compose.yml         # NEW
└── TODO.md                    # this file
```

---

## Tasks

### 1. Copy source trees

- [ ] `cp -r ../backend v3/backend`
- [ ] `cp -r ../ui-v2 v3/ui`
- [ ] `cp ../scripts/pb-init-v2.sh v3/scripts/pb-init-v3.sh`
- [ ] Verify `v3/backend/go.mod` module path is still `github.com/libnovel/backend`
  (no rename needed — it's a copy, not a Go workspace dep)

### 2. Add Valkey presign cache (`v3/backend`)

**Goal**: Backend generates presigned URLs → stores in Valkey with TTL.
UI reads from Valkey before calling backend.

Files to create/modify:

- [ ] `internal/presigncache/cache.go` — `Cache` interface + `ValkeyCache` implementation
  - `Get(ctx, key) (string, bool, error)`
  - `Set(ctx, key, url string, ttl time.Duration) error`
  - Uses `github.com/redis/go-redis/v9`
  - TTL: match presigned URL lifetime (e.g. 45 min; MinIO default is 1h → use 55 min)
- [ ] `internal/config/config.go` — add `ValkeyAddr string` (default `"valkey:6379"`)
- [ ] `internal/backend/server.go` — add `PresignCache presigncache.Cache` to `Dependencies`
- [ ] `internal/backend/handlers.go` — in presign handlers, do `cache.Get` before generating,
  `cache.Set` after generating
- [ ] `go.mod` — add `github.com/redis/go-redis/v9`
- [ ] `cmd/backend/main.go` — wire `ValkeyCache` from config

### 3. Add Meilisearch integration (`v3/backend`)

**Goal**: On each book scrape completion, upsert the book document into Meilisearch.
Backend `/api/search` returns Meilisearch results for local books, falling back to
novelfire.net live search for books not in the index.

Files to create/modify:

- [ ] `internal/meili/client.go` — `Client` interface + `MeiliClient` implementation
  - `UpsertBook(ctx, book domain.Book) error`
  - `Search(ctx, query string, limit int) ([]domain.Book, error)`
  - Uses `github.com/meilisearch/meilisearch-go`
  - Index name: `books`, primary key: `slug`
  - Searchable attributes: `title`, `author`, `tags`, `description`
  - Filterable attributes: `status`, `tags`
- [ ] `internal/config/config.go` — add `MeiliURL string`, `MeiliAPIKey string`
- [ ] `internal/bookstore/bookstore.go` — add `SearchIndex` interface with `UpsertBook`
- [ ] `internal/runner/runner.go` — inject `SearchIndex`; after `FinishScrapeTask` success,
  call `SearchIndex.UpsertBook` for each book scraped
- [ ] `internal/backend/server.go` — add `SearchIndex` to `Dependencies`; register `GET /api/search`
- [ ] `internal/backend/handlers.go` — `searchHandler`: query Meilisearch first; if 0 results,
  fall back to novelfire.net live search via `NovelScraper.SearchBooks`
- [ ] `go.mod` — add `github.com/meilisearch/meilisearch-go`
- [ ] `cmd/backend/main.go` + `cmd/runner/main.go` — wire `MeiliClient`

### 4. Add runner `/metrics` HTTP endpoint (`v3/backend`)

**Goal**: Runner exposes `GET /metrics` returning JSON with task counters. Caddy can
healthcheck this; operators can scrape it from a monitoring tool.

Payload example:
```json
{
  "tasks_pending":    0,
  "tasks_running":    1,
  "tasks_completed": 42,
  "tasks_failed":     2,
  "uptime_seconds": 3600
}
```

Files to create/modify:

- [ ] `internal/runner/metrics.go` — `Metrics` struct with `atomic.Int64` counters;
  `ServeHTTP` handler returns JSON; `MetricsServer` starts `net/http` on configurable addr
- [ ] `internal/runner/runner.go` — embed `*Metrics`; increment counters at task lifecycle
  points (claimed → running → completed/failed)
- [ ] `internal/config/config.go` — add `RunnerMetricsAddr string` (default `":9091"`)
- [ ] `cmd/runner/main.go` — start `MetricsServer` in background goroutine before `runner.Run`
- [ ] `docker-compose.yml` (runner service) — expose port 9091 internally; update healthcheck
  to `GET http://localhost:9091/metrics` (replaces file-based liveness)

### 5. Update UI presign cache to use Valkey (`v3/ui`)

**Goal**: Replace `src/lib/server/presignCache.ts` (module-scope `Map`) with Valkey via
`ioredis`. Cache survives UI restarts and is shared if multiple UI replicas run.

Files to modify:

- [ ] `src/lib/server/presignCache.ts` — rewrite to use `ioredis` client
  - `getPresignedUrl(key: string): Promise<string | null>`
  - `setPresignedUrl(key: string, url: string, ttlSeconds: number): Promise<void>`
  - Connection string from `VALKEY_URL` env var (default `redis://valkey:6379`)
  - Keep the same TTL logic (50 min) and sweep logic (not needed — Valkey TTL is native)
- [ ] `package.json` — add `ioredis` (or use `@redis/client` — prefer `ioredis` for its
  robust reconnection handling)
- [ ] Any callers of old `presignCache` — update import if interface changes

### 6. Write `v3/Caddyfile`

- [ ] Global options block: email for ACME, staging CA override for local dev
- [ ] Single site block matching `{$DOMAIN}` (env var injection):
  - `handle /api/*` → `reverse_proxy backend:8080`
  - `handle /health` → `reverse_proxy backend:8080`
  - `handle /s3/*` → strip prefix + `reverse_proxy minio:9000` (internal bucket access,
    requires `Authorization` passthrough)
  - `handle` (catch-all) → `reverse_proxy ui:3000`
- [ ] Health/liveness endpoints exposed without auth
- [ ] No ports on MinIO/PocketBase exposed to public (internal Docker network only)

### 7. Write `v3/docker-compose.yml`

Services (in dependency order):

| Service | Image / Build | Ports (host) | Notes |
|---------|--------------|--------------|-------|
| `minio` | `minio/minio:latest` | none (internal only) | Remove public port exposure; Caddy proxies `/s3/*` |
| `minio-init` | `minio/mc:latest` | — | Same as v2 |
| `pocketbase` | `ghcr.io/muchobien/pocketbase:latest` | none (internal only) | Remove public port |
| `pb-init` | `alpine:3.19` | — | Same as v2, uses `v3/scripts/pb-init-v3.sh` |
| `valkey` | `valkey/valkey:8-alpine` | none (internal) | `valkey-server --save "" --appendonly no` |
| `meilisearch` | `getmeili/meilisearch:v1.7` | none (internal) | `MEILI_NO_ANALYTICS=true`, persistent volume |
| `backend` | build `v3/backend` target `backend` | none (internal) | Add `VALKEY_ADDR`, `MEILI_URL`, `MEILI_API_KEY` |
| `runner` | build `v3/backend` target `runner` | `9091` (metrics) | Add `RUNNER_METRICS_ADDR`, `MEILI_URL`, `MEILI_API_KEY`; healthcheck via `/metrics` |
| `ui` | build `v3/ui` | none (internal) | Add `VALKEY_URL`; remove `PUBLIC_MINIO_PUBLIC_URL` (all through Caddy now) |
| `caddy` | `caddy:2-alpine` | `80`, `443` | Bind-mount `v3/Caddyfile`; persistent `caddy_data` volume for certs |

Volumes: `minio_data`, `pb_data`, `meili_data`, `caddy_data`

- [ ] Write the full file with healthchecks, depends_on, env vars

### 8. Write `v3/scripts/pb-init-v3.sh`

- [ ] Start from `scripts/pb-init-v2.sh`
- [ ] Assess if any schema changes needed (Meilisearch sync does not require new PB collections
  — search is entirely Meilisearch-side)
- [ ] If identical to v2 script, note that in a comment at the top; keep as separate file

---

## Environment variables (new in v3)

| Variable | Service | Default | Description |
|----------|---------|---------|-------------|
| `VALKEY_ADDR` | backend | `valkey:6379` | Valkey TCP address |
| `VALKEY_URL` | ui | `redis://valkey:6379` | Valkey URL (ioredis format) |
| `MEILI_URL` | backend, runner | `http://meilisearch:7700` | Meilisearch HTTP URL |
| `MEILI_API_KEY` | backend, runner | `""` | Meilisearch master key (empty = no auth for dev) |
| `RUNNER_METRICS_ADDR` | runner | `:9091` | Runner metrics HTTP listen address |
| `DOMAIN` | caddy | `localhost` | Public domain for HTTPS cert |
| `CADDY_ACME_EMAIL` | caddy | `""` | Let's Encrypt notification email |

---

## Testing checklist (before considering v3 stable)

- [ ] `docker compose -f v3/docker-compose.yml up --build` starts cleanly
- [ ] `curl https://{DOMAIN}/health` returns 200 (Caddy → backend)
- [ ] Book search returns results from Meilisearch after at least one scrape
- [ ] Presign URL cache hit visible in backend logs (Valkey `GET` hit)
- [ ] `curl http://localhost:9091/metrics` returns JSON with counters
- [ ] Runner healthcheck passes via `/metrics` endpoint (not file-based)
- [ ] MinIO console NOT reachable from the internet (no public port)
- [ ] PocketBase NOT reachable from the internet (no public port)
- [ ] TLS cert obtained from Let's Encrypt (check `caddy_data` volume / Caddy logs)