libnovel/v3/TODO.md

v3 Implementation Plan

v3 is a self-contained directory. All services live under v3/. Nothing shares code with the root backend/ or ui-v2/ directories.

New in v3 vs v2

Addition	Why
Caddy reverse proxy	Single public entry point, automatic HTTPS via Let's Encrypt, no manual cert management
Meilisearch	Full-text search over locally scraped books; replaces live novelfire.net scrape on every search
Valkey (Redis-compatible)	Shared presign URL cache; replaces the per-process in-memory Map in the UI Node process
Runner /metrics HTTP endpoint	Exposes task counters (pending, running, failed, completed) for healthcheck / Caddy visibility

Directory layout (target)

v3/
├── backend/                   # copied from ../backend/, then modified
│   ├── cmd/
│   │   ├── backend/           # unchanged
│   │   └── runner/            # add metrics HTTP server
│   ├── internal/
│   │   ├── backend/
│   │   │   ├── server.go      # add /api/search endpoint (Meilisearch)
│   │   │   └── handlers.go    # add searchHandler, presign now reads Valkey
│   │   ├── runner/
│   │   │   ├── runner.go      # add atomic counters + metrics HTTP server
│   │   │   └── metrics.go     # NEW: MetricsServer (net/http, /metrics JSON)
│   │   ├── meili/             # NEW: Meilisearch client package
│   │   │   └── client.go
│   │   ├── presigncache/      # NEW: Valkey-backed presign cache
│   │   │   └── cache.go
│   │   └── config/
│   │       └── config.go      # add Meilisearch + Valkey + metrics addr vars
│   └── go.mod                 # add meilisearch-go, go-redis/v9
├── ui/                        # copied from ../ui-v2/, then modified
│   └── src/lib/server/
│       └── presignCache.ts    # replace in-process Map with Valkey (ioredis)
├── docs/
│   ├── architecture.d2        # DONE
│   └── architecture.mermaid.md # DONE
├── scripts/
│   └── pb-init-v3.sh          # same as v2 unless schema changes
├── Caddyfile                  # NEW
├── docker-compose.yml         # NEW
└── TODO.md                    # this file

---

Tasks

1. Copy source trees

• cp -r ../backend v3/backend
• cp -r ../ui-v2 v3/ui
• cp ../scripts/pb-init-v2.sh v3/scripts/pb-init-v3.sh
• Verify v3/backend/go.mod module path is still github.com/libnovel/backend (no rename needed — it's a copy, not a Go workspace dep)

2. Add Valkey presign cache (v3/backend)

Goal: Backend generates presigned URLs → stores in Valkey with TTL. UI reads from Valkey before calling backend.

Files to create/modify:

• internal/presigncache/cache.go — Cache interface + ValkeyCache implementation
  • Get(ctx, key) (string, bool, error)
  • Set(ctx, key, url string, ttl time.Duration) error
  • Uses github.com/redis/go-redis/v9
  • TTL: match presigned URL lifetime (e.g. 45 min; MinIO default is 1h → use 55 min)
• internal/config/config.go — add ValkeyAddr string (default "valkey:6379")
• internal/backend/server.go — add PresignCache presigncache.Cache to Dependencies
• internal/backend/handlers.go — in presign handlers, do cache.Get before generating, cache.Set after generating
• go.mod — add github.com/redis/go-redis/v9
• cmd/backend/main.go — wire ValkeyCache from config

3. Add Meilisearch integration (v3/backend)

Goal: On each book scrape completion, upsert the book document into Meilisearch. Backend /api/search returns Meilisearch results for local books, falling back to novelfire.net live search for books not in the index.

Files to create/modify:

• internal/meili/client.go — Client interface + MeiliClient implementation
  • UpsertBook(ctx, book domain.Book) error
  • Search(ctx, query string, limit int) ([]domain.Book, error)
  • Uses github.com/meilisearch/meilisearch-go
  • Index name: books, primary key: slug
  • Searchable attributes: title, author, tags, description
  • Filterable attributes: status, tags
• internal/config/config.go — add MeiliURL string, MeiliAPIKey string
• internal/bookstore/bookstore.go — add SearchIndex interface with UpsertBook
• internal/runner/runner.go — inject SearchIndex; after FinishScrapeTask success, call SearchIndex.UpsertBook for each book scraped
• internal/backend/server.go — add SearchIndex to Dependencies; register GET /api/search
• internal/backend/handlers.go — searchHandler: query Meilisearch first; if 0 results, fall back to novelfire.net live search via NovelScraper.SearchBooks
• go.mod — add github.com/meilisearch/meilisearch-go
• cmd/backend/main.go + cmd/runner/main.go — wire MeiliClient

4. Add runner /metrics HTTP endpoint (v3/backend)

Goal: Runner exposes GET /metrics returning JSON with task counters. Caddy can healthcheck this; operators can scrape it from a monitoring tool.

Payload example:

{
  "tasks_pending":    0,
  "tasks_running":    1,
  "tasks_completed": 42,
  "tasks_failed":     2,
  "uptime_seconds": 3600
}

Files to create/modify:

• internal/runner/metrics.go — Metrics struct with atomic.Int64 counters; ServeHTTP handler returns JSON; MetricsServer starts net/http on configurable addr
• internal/runner/runner.go — embed *Metrics; increment counters at task lifecycle points (claimed → running → completed/failed)
• internal/config/config.go — add RunnerMetricsAddr string (default ":9091")
• cmd/runner/main.go — start MetricsServer in background goroutine before runner.Run
• docker-compose.yml (runner service) — expose port 9091 internally; update healthcheck to GET http://localhost:9091/metrics (replaces file-based liveness)

5. Update UI presign cache to use Valkey (v3/ui)

Goal: Replace src/lib/server/presignCache.ts (module-scope Map) with Valkey via ioredis. Cache survives UI restarts and is shared if multiple UI replicas run.

Files to modify:

• src/lib/server/presignCache.ts — rewrite to use ioredis client
  • getPresignedUrl(key: string): Promise<string | null>
  • setPresignedUrl(key: string, url: string, ttlSeconds: number): Promise<void>
  • Connection string from VALKEY_URL env var (default redis://valkey:6379)
  • Keep the same TTL logic (50 min) and sweep logic (not needed — Valkey TTL is native)
• package.json — add ioredis (or use @redis/client — prefer ioredis for its robust reconnection handling)
• Any callers of old presignCache — update import if interface changes

6. Write v3/Caddyfile

• Global options block: email for ACME, staging CA override for local dev
• Single site block matching {$DOMAIN} (env var injection):
  • handle /api/* → reverse_proxy backend:8080
  • handle /health → reverse_proxy backend:8080
  • handle /s3/* → strip prefix + reverse_proxy minio:9000 (internal bucket access, requires Authorization passthrough)
  • handle (catch-all) → reverse_proxy ui:3000
• Health/liveness endpoints exposed without auth
• No ports on MinIO/PocketBase exposed to public (internal Docker network only)

7. Write v3/docker-compose.yml

Services (in dependency order):

Service	Image / Build	Ports (host)	Notes
minio	minio/minio:latest	none (internal only)	Remove public port exposure; Caddy proxies /s3/*
minio-init	minio/mc:latest	—	Same as v2
pocketbase	ghcr.io/muchobien/pocketbase:latest	none (internal only)	Remove public port
pb-init	alpine:3.19	—	Same as v2, uses v3/scripts/pb-init-v3.sh
valkey	valkey/valkey:8-alpine	none (internal)	valkey-server --save "" --appendonly no
meilisearch	getmeili/meilisearch:v1.7	none (internal)	MEILI_NO_ANALYTICS=true, persistent volume
backend	build v3/backend target backend	none (internal)	Add VALKEY_ADDR, MEILI_URL, MEILI_API_KEY
runner	build v3/backend target runner	9091 (metrics)	Add RUNNER_METRICS_ADDR, MEILI_URL, MEILI_API_KEY; healthcheck via /metrics
ui	build v3/ui	none (internal)	Add VALKEY_URL; remove PUBLIC_MINIO_PUBLIC_URL (all through Caddy now)
caddy	caddy:2-alpine	80, 443	Bind-mount v3/Caddyfile; persistent caddy_data volume for certs

Volumes: minio_data, pb_data, meili_data, caddy_data

• Write the full file with healthchecks, depends_on, env vars

8. Write v3/scripts/pb-init-v3.sh

• Start from scripts/pb-init-v2.sh
• Assess if any schema changes needed (Meilisearch sync does not require new PB collections — search is entirely Meilisearch-side)
• If identical to v2 script, note that in a comment at the top; keep as separate file

---

Environment variables (new in v3)

Variable	Service	Default	Description
VALKEY_ADDR	backend	valkey:6379	Valkey TCP address
VALKEY_URL	ui	redis://valkey:6379	Valkey URL (ioredis format)
MEILI_URL	backend, runner	http://meilisearch:7700	Meilisearch HTTP URL
MEILI_API_KEY	backend, runner	""	Meilisearch master key (empty = no auth for dev)
RUNNER_METRICS_ADDR	runner	:9091	Runner metrics HTTP listen address
DOMAIN	caddy	localhost	Public domain for HTTPS cert
CADDY_ACME_EMAIL	caddy	""	Let's Encrypt notification email

---

Testing checklist (before considering v3 stable)

• docker compose -f v3/docker-compose.yml up --build starts cleanly
• curl https://{DOMAIN}/health returns 200 (Caddy → backend)
• Book search returns results from Meilisearch after at least one scrape
• Presign URL cache hit visible in backend logs (Valkey GET hit)
• curl http://localhost:9091/metrics returns JSON with counters
• Runner healthcheck passes via /metrics endpoint (not file-based)
• MinIO console NOT reachable from the internet (no public port)
• PocketBase NOT reachable from the internet (no public port)
• TLS cert obtained from Let's Encrypt (check caddy_data volume / Caddy logs)