chore: migrate to v3, Doppler secrets, clean up legacy code

- Remove all pre-v3 code: scraper, ui-v2, backend v1, ios v1+v2, legacy CI workflows - Flatten v3/ contents to repo root - Add Doppler secrets management (project=libnovel, config=prd) - Add justfile with doppler run wrappers for all docker compose commands - Strip hardcoded env fallbacks from docker-compose.yml - Add minimal README.md - Clean up .gitignore
2026-03-23 17:21:12 +05:00
parent 1118392811
commit 59e8cdb19a
522 changed files with 5259 additions and 80365 deletions
--- a/docs/mermaid/architecture.mermaid.md
+++ b/docs/mermaid/architecture.mermaid.md
@@ -0,0 +1,72 @@
+# Architecture Overview
+
+```mermaid
+graph LR
+    %% ── External ──────────────────────────────────────────────────────────
+    NF([novelfire.net])
+    KK([Kokoro-FastAPI TTS])
+    LE([Let's Encrypt])
+    CL([Browser / iOS App])
+
+    %% ── Init containers ───────────────────────────────────────────────────
+    subgraph INIT["Init containers (one-shot)"]
+        MI[minio-init\nmc: create buckets]
+        PI[pb-init\nbootstrap collections]
+    end
+
+    %% ── Storage ───────────────────────────────────────────────────────────
+    subgraph STORAGE["Storage"]
+        MN[(MinIO :9000\nchapters · audio\navatars · browse)]
+        PB[(PocketBase :8090\nbooks · chapters_idx\naudio_cache · progress\nscrape_jobs · app_users · ranking)]
+        VK[(Valkey :6379\npresign URL cache\nTTL-based · shared)]
+        MS[(Meilisearch :7700\nindex: books)]
+    end
+
+    %% ── Application ───────────────────────────────────────────────────────
+    subgraph APP["Application"]
+        CD["Caddy :443/:80\ncustom build + caddy-ratelimit\nauto-HTTPS · security headers\nrate limiting · error pages"]
+        BE[Backend API :8080\nGo HTTP server]
+        RN[Runner :9091\nGo background worker\n/metrics endpoint]
+        UI[SvelteKit UI :3000\nadapter-node]
+    end
+
+    %% ── Ops ───────────────────────────────────────────────────────────────
+    subgraph OPS["Ops"]
+        WT[Watchtower\npolls every 5 min\nautopull + redeploy\nbackend · runner · ui]
+    end
+
+    %% ── Init → Storage ────────────────────────────────────────────────────
+    MI -.->|create buckets| MN
+    PI -.->|bootstrap schema| PB
+
+    %% ── App → Storage ─────────────────────────────────────────────────────
+    BE -->|blobs| MN
+    BE -->|structured records| PB
+    BE -->|cache presigned URLs| VK
+    RN -->|chapter markdown & audio| MN
+    RN -->|read/update jobs & books| PB
+    RN -->|index books on scrape| MS
+    UI -->|read presign cache| VK
+    UI -->|auth · progress · comments| PB
+
+    %% ── App internal ──────────────────────────────────────────────────────
+    UI -->|"REST API (server-side)\n/api/catalogue /api/book-preview\n/api/chapter-text /api/audio"| BE
+
+    %% ── Caddy routing ─────────────────────────────────────────────────────
+    CD -->|"/* catch-all\n/api/scrape/*\n/api/chapter-text-preview/*\n→ SvelteKit (auth enforced)"| UI
+    CD -->|"/health /scrape*\n/api/browse /api/book-preview/*\n/api/chapter-text/*\n/api/reindex/* /api/cover/*\n/api/audio-proxy/*"| BE
+    CD -->|/avatars/* presigned GETs| MN
+
+    %% ── Runner → External ─────────────────────────────────────────────────
+    RN -->|scrape HTTP GET| NF
+    RN -->|TTS HTTP POST| KK
+    CD -->|ACME certificate| LE
+
+    %% ── Ops ───────────────────────────────────────────────────────────────
+    WT -->|watch label-enabled| BE
+    WT -->|watch label-enabled| RN
+    WT -->|watch label-enabled| UI
+
+    %% ── Client ────────────────────────────────────────────────────────────
+    CL -->|HTTPS :443 single entry| CD
+```
--- a/docs/mermaid/data-flow.mermaid.md
+++ b/docs/mermaid/data-flow.mermaid.md
@@ -0,0 +1,102 @@
+# Data Flow — Scrape & TTS Job Pipeline
+
+How content moves from novelfire.net through the runner into storage, and how
+audio is generated on-demand via the backend.
+
+## Catalogue Scrape Pipeline
+
+The runner performs a background catalogue walk on startup and then on a
+configurable interval (`RUNNER_CATALOGUE_REFRESH_INTERVAL`, default 24 h).
+
+```mermaid
+flowchart TD
+    A([Runner starts / refresh tick]) --> B[Walk novelfire.net catalogue\npages 1…N]
+    B --> C{Book already\nin PocketBase?}
+    C -- no --> D[Scrape book metadata\ntitle · author · genres\ncover · summary · status]
+    C -- yes --> E[Check for new chapters\ncompare total_chapters]
+    D --> F[Write BookMeta\nto PocketBase books]
+    E --> G{New chapters\nfound?}
+    G -- no --> Z([Done — next book])
+    G -- yes --> H
+    F --> H[Scrape chapter list with upTo limit\n→ chapters_idx in PocketBase\nretries on 429 with Retry-After backoff]
+    H --> I[Worker pool — N goroutines\nRUNNER_MAX_CONCURRENT_SCRAPE]
+    I --> J[For each missing chapter:\nGET chapter HTML from novelfire.net]
+    J --> K[Parse HTML → Markdown\nhtmlutil.NodeToMarkdown]
+    K --> L[PUT object to MinIO\nchapters/{slug}/{n}.md]
+    L --> M[Upsert book doc\nto Meilisearch index: books]
+    M --> Z
+    F --> M
+```
+
+## On-Demand Single-Book Scrape
+
+Triggered when a user visits `/books/{slug}` and the book is not in PocketBase.
+The UI calls `GET /api/book-preview/{slug}` → backend enqueues a scrape task.
+
+```mermaid
+sequenceDiagram
+    actor U as User
+    participant UI as SvelteKit UI
+    participant BE as Backend API
+    participant TQ as Task Queue (PocketBase)
+    participant RN as Runner
+    participant NF as novelfire.net
+    participant PB as PocketBase
+    participant MN as MinIO
+    participant MS as Meilisearch
+
+    U->>UI: Visit /books/{slug}
+    UI->>BE: GET /api/book-preview/{slug}
+    BE->>PB: getBook(slug) — not found
+    BE->>TQ: INSERT scrape_task (slug, status=pending)
+    BE-->>UI: 202 {task_id, message}
+    UI-->>U: "Scraping…" placeholder
+
+    RN->>TQ: Poll for pending tasks
+    TQ-->>RN: scrape_task (slug)
+    RN->>NF: GET novelfire.net/book/{slug}
+    NF-->>RN: HTML
+    RN->>PB: upsert book + chapters_idx
+    RN->>MN: PUT chapter objects
+    RN->>MS: UpsertBook doc
+    RN->>TQ: UPDATE task status=done
+
+    U->>UI: Poll GET /api/scrape/tasks/{task_id}
+    UI->>BE: GET /api/scrape/status
+    BE->>TQ: get task
+    TQ-->>BE: status=done
+    BE-->>UI: {status:"done"}
+    UI-->>U: Redirect to /books/{slug}
+```
+
+## TTS Audio Generation Pipeline
+
+Audio is generated lazily: on first request the job is enqueued; subsequent
+requests poll for completion and then stream from MinIO via presigned URL.
+
+```mermaid
+flowchart TD
+    A(["POST /api/audio/{slug}/{n}\nbody: voice=af_bella"]) --> B{Audio already\nin MinIO?}
+    B -- yes --> C[200 status: done]
+    B -- no --> D{Job already\nin queue?}
+    D -- "yes pending/generating" --> E[202 task_id + status]
+    D -- no --> F[INSERT audio_task\nstatus=pending\nin PocketBase]
+    F --> E
+
+    G([Runner polls task queue]) --> H[Claim audio_task\nstatus=generating]
+    H --> I["GET /api/chapter-text/{slug}/{n}\nfrom backend — plain text"]
+    I --> J[POST /v1/audio/speech\nto Kokoro-FastAPI\nbody: text + voice]
+    J --> K[Stream MP3 response]
+    K --> L[PUT object to MinIO\naudio/{slug}/{n}/{voice}.mp3]
+    L --> M[UPDATE audio_task\nstatus=done]
+
+    N(["Client polls\nGET /api/audio/status/{slug}/{n}"]) --> O{status?}
+    O -- "pending/generating" --> N
+    O -- done --> P["GET /api/presign/audio/{slug}/{n}"]
+    P --> Q{Valkey cache hit?}
+    Q -- yes --> R[302 → presigned URL]
+    Q -- no --> S[GeneratePresignedURL\nfrom MinIO — TTL 1h]
+    S --> T[Cache in Valkey\nTTL 3500s]
+    T --> R
+    R --> U([Client streams audio\ndirectly from MinIO])
+```
--- a/docs/mermaid/request-flow.mermaid.md
+++ b/docs/mermaid/request-flow.mermaid.md
@@ -0,0 +1,111 @@
+# Request Flow
+
+Two representative request paths through the stack: a **page load** (SSR) and a
+**media playback** (presigned URL → direct MinIO stream).
+
+## SSR Page Load — Catalogue / Book Detail
+
+```mermaid
+sequenceDiagram
+    actor C as Browser / iOS App
+    participant CD as Caddy :443
+    participant UI as SvelteKit UI :3000
+    participant BE as Backend API :8080
+    participant MS as Meilisearch :7700
+    participant PB as PocketBase :8090
+    participant VK as Valkey :6379
+
+    C->>CD: HTTPS GET /catalogue
+    CD->>UI: proxy /* (SvelteKit catch-all)
+    UI->>BE: GET /api/catalogue?page=1&sort=popular
+    BE->>MS: search(query, filters, sort)
+    MS-->>BE: [{slug, title, …}, …]
+    BE-->>UI: {books[], page, total, has_next}
+    UI-->>CD: SSR HTML
+    CD-->>C: 200 HTML
+
+    Note over C,UI: Infinite scroll — client fetches next page via SvelteKit API route
+    C->>CD: HTTPS GET /api/catalogue-page?page=2
+    CD->>UI: proxy /* (SvelteKit /api/catalogue-page server route)
+    UI->>BE: GET /api/catalogue?page=2
+    BE->>MS: search(…)
+    MS-->>BE: next page
+    BE-->>UI: {books[], …}
+    UI-->>C: JSON
+```
+
+## Audio Playback — Presigned URL Flow
+
+```mermaid
+sequenceDiagram
+    actor C as Browser / iOS App
+    participant CD as Caddy :443
+    participant UI as SvelteKit UI :3000
+    participant BE as Backend API :8080
+    participant VK as Valkey :6379
+    participant MN as MinIO :9000
+
+    C->>CD: GET /api/presign/audio/{slug}/{n}?voice=af_bella
+    CD->>UI: proxy /* (SvelteKit /api/presign/audio route)
+    UI->>BE: GET /api/presign/audio/{slug}/{n}?voice=af_bella
+    BE->>VK: GET presign:audio:{slug}:{n}:{voice}
+    alt cache hit
+        VK-->>BE: presigned URL (TTL remaining)
+        BE-->>UI: 302 redirect → presigned URL
+        UI-->>C: 302 redirect
+    else cache miss
+        BE->>MN: GeneratePresignedURL(audio-bucket, key, 1h)
+        MN-->>BE: presigned URL
+        BE->>VK: SET presign:audio:… EX 3500
+        BE-->>UI: 302 redirect → presigned URL
+        UI-->>C: 302 redirect
+    end
+    C->>MN: GET presigned URL (direct, no proxy)
+    MN-->>C: audio/mpeg stream
+```
+
+## Chapter Read — SSR + Content Fetch
+
+```mermaid
+sequenceDiagram
+    actor C as Browser / iOS App
+    participant CD as Caddy :443
+    participant UI as SvelteKit UI :3000
+    participant BE as Backend API :8080
+    participant PB as PocketBase :8090
+    participant MN as MinIO :9000
+
+    C->>CD: HTTPS GET /books/{slug}/chapters/{n}
+    CD->>UI: proxy /* (SvelteKit catch-all)
+    UI->>PB: getBook(slug) + listChapterIdx(slug)
+    PB-->>UI: book meta + chapter list
+    UI->>BE: GET /api/chapter-text/{slug}/{n}
+    BE->>MN: GetObject(chapters-bucket, {slug}/{n}.md)
+    MN-->>BE: markdown text
+    BE-->>UI: plain text (markdown stripped)
+    Note over UI: marked() → HTML
+    UI-->>CD: SSR HTML
+    CD-->>C: 200 HTML
+```
+
+## Caddy Request Lifecycle
+
+Shows how security hardening applies before a request reaches any upstream.
+
+```mermaid
+flowchart TD
+    A([Incoming HTTPS request]) --> B[TLS termination\nLet's Encrypt cert]
+    B --> C{Rate limit check\ncaddy-ratelimit}
+    C -- over limit --> D[429 Too Many Requests]
+    C -- ok --> E[Add security headers\nX-Frame-Options · X-Content-Type-Options\nReferrer-Policy · Permissions-Policy\nHSTS · X-XSS-Protection\nremove Server header]
+    E --> F{Route match}
+    F -- "/health /scrape*\n/api/browse /api/book-preview/*\n/api/chapter-text/*\n/api/reindex/* /api/cover/*\n/api/audio-proxy/*" --> G[reverse_proxy → backend:8080]
+    F -- "/avatars/*" --> H[reverse_proxy → minio:9000]
+    F -- "/* everything else\n(incl. /api/scrape/*\n/api/chapter-text-preview/*)" --> I[reverse_proxy → ui:3000\nSvelteKit auth middleware runs]
+    G --> J{Upstream healthy?}
+    H --> J
+    I --> J
+    J -- yes --> K([Response to client])
+    J -- "502/503/504" --> L[handle_errors\nstatic HTML from /srv/errors/]
+    L --> K
+```