feat(v3): add v3 stack — backend rewrite, renamed env vars, docs

- New Go backend binary (backend + runner) replacing old scraper/
- Rename SCRAPER_API_URL → BACKEND_API_URL in UI env and docker-compose
- Rename scraperFetch → backendFetch across all 19 UI server files
- Remove SCRAPER_PROXY env var and proxy transport from browser.Config
- Add Meilisearch, Valkey, Caddy to docker-compose
- Add docs/: api-endpoints.md, request-flow.mermaid.md, data-flow.mermaid.md
This commit is contained in:
Admin
2026-03-22 17:27:32 +05:00
parent 29d0eeb7e8
commit a85636d5db
178 changed files with 25951 additions and 0 deletions

82
v3/docs/api-endpoints.md Normal file
View File

@@ -0,0 +1,82 @@
# API Endpoint Reference
All endpoints served by the Go **backend** binary on `:8080`. In production all
traffic is routed through Caddy — `/api/*` and `/health` are proxied to the
backend; everything else goes to the SvelteKit UI.
## Health / Version
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/health` | — | Liveness probe. Returns `{"ok":true}`. |
| `GET` | `/api/version` | — | Build version + commit hash. |
## Scrape Jobs (admin)
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `POST` | `/scrape` | admin | Enqueue full catalogue scrape. |
| `POST` | `/scrape/book` | admin | Enqueue single-book scrape `{url}`. |
| `POST` | `/scrape/book/range` | admin | Enqueue range scrape `{url, from, to?}`. |
| `GET` | `/api/scrape/status` | admin | Current job status. |
| `GET` | `/api/scrape/tasks` | admin | All scrape task records. |
| `POST` | `/api/cancel-task/{id}` | admin | Cancel a pending task. |
## Browse / Catalogue
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/api/browse` | — | Live novelfire.net browse (MinIO page-1 cache). Legacy — used by save-browse subcommand. |
| `GET` | `/api/catalogue` | — | **Primary browse endpoint.** Meilisearch-backed, paginated. Params: `q`, `page`, `limit`, `genre`, `status`, `sort` (`popular`\|`new`\|`update`\|`rank`\|`top-rated`). Falls back to empty when Meilisearch is not configured. |
| `GET` | `/api/search` | — | Full-text search: Meilisearch local results merged with live novelfire.net remote results. Param: `q` (≥ 2 chars). Used by iOS app. |
| `GET` | `/api/ranking` | — | Top-ranked novels from PocketBase. |
| `GET` | `/api/cover/{domain}/{slug}` | — | Proxy cover image from MinIO (redirect to presigned URL). |
## Book / Chapter Content
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/api/book-preview/{slug}` | — | Returns stored metadata + chapter list, or enqueues a scrape task (202) if unknown. |
| `GET` | `/api/chapter-text/{slug}/{n}` | — | Chapter content as plain text (markdown stripped). |
| `GET` | `/api/chapter-markdown/{slug}/{n}` | — | Chapter content as raw markdown from MinIO. |
| `POST` | `/api/reindex/{slug}` | admin | Rebuild `chapters_idx` from MinIO objects. |
## Audio
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `POST` | `/api/audio/{slug}/{n}` | — | Trigger Kokoro TTS generation. Body: `{voice?}`. Returns `200 {status:"done"}` if cached, `202 {task_id, status}` if enqueued. |
| `GET` | `/api/audio/status/{slug}/{n}` | — | Poll audio generation status. Param: `voice`. Returns `{status, task_id?, error?}`. |
| `GET` | `/api/audio-proxy/{slug}/{n}` | — | Redirect to presigned MinIO audio URL. |
| `GET` | `/api/voices` | — | List available Kokoro voices. Returns `{voices:[]}` on error. |
## Presigned URLs
All presign endpoints return a `302` redirect to a short-lived MinIO presigned
URL. The URL is cached in Valkey (TTL ~55 min) to avoid regenerating on every
request.
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/api/presign/chapter/{slug}/{n}` | — | Presigned URL for chapter markdown object. |
| `GET` | `/api/presign/audio/{slug}/{n}` | — | Presigned URL for audio MP3. Param: `voice`. |
| `GET` | `/api/presign/voice-sample/{voice}` | — | Presigned URL for voice sample MP3. |
| `GET` | `/api/presign/avatar-upload/{userId}` | user | Presigned PUT URL for avatar upload. |
| `GET` | `/api/presign/avatar/{userId}` | — | Presigned GET URL for avatar image. |
## Reading Progress
Session-scoped (anonymous via cookie session ID, or tied to authenticated user).
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/api/progress` | — | Get all reading progress for the current session/user. |
| `POST` | `/api/progress/{slug}` | — | Set progress. Body: `{chapter}`. |
| `DELETE` | `/api/progress/{slug}` | — | Delete progress for a book. |
## Notes
- **Auth**: The backend does not enforce auth itself — the SvelteKit UI layer enforces admin/user guards before proxying requests. The backend trusts all incoming requests.
- **`/api/catalogue` vs `/api/browse`**: `/api/catalogue` is the primary UI endpoint (Meilisearch, always-local, fast). `/api/browse` hits or caches the live novelfire.net browse page and is only used internally by the `save-browse` subcommand.
- **Meilisearch fallback**: When `MEILI_URL` is unset, `/api/catalogue` returns `{books:[], has_next:false}` and `/api/search` falls back to a PocketBase substring scan.
- **`BACKEND_API_URL`**: The SvelteKit UI reads this env var (default `http://localhost:8080`) to reach the backend server-side. In docker-compose it is set to `http://backend:8080`.

129
v3/docs/architecture.d2 Normal file
View File

@@ -0,0 +1,129 @@
direction: right
# ─── External ─────────────────────────────────────────────────────────────────
novelfire: novelfire.net {
shape: cloud
style.fill: "#f0f4ff"
}
kokoro: Kokoro-FastAPI TTS {
shape: cloud
style.fill: "#f0f4ff"
}
letsencrypt: Let's Encrypt {
shape: cloud
style.fill: "#f0f4ff"
}
browser: Browser / iOS App {
shape: person
style.fill: "#fff9e6"
}
# ─── Init containers (one-shot) ───────────────────────────────────────────────
init: Init containers {
style.fill: "#f5f5f5"
style.stroke-dash: 4
minio-init: minio-init {
shape: rectangle
label: "minio-init\n(mc: create buckets)"
}
pb-init: pb-init {
shape: rectangle
label: "pb-init\n(bootstrap collections)"
}
}
# ─── Storage ──────────────────────────────────────────────────────────────────
storage: Storage {
style.fill: "#eaf7ea"
minio: MinIO {
shape: cylinder
label: "MinIO :9000\n\nbuckets:\n libnovel-chapters\n libnovel-audio\n libnovel-avatars\n libnovel-browse"
}
pocketbase: PocketBase {
shape: cylinder
label: "PocketBase :8090\n\ncollections:\n books chapters_idx\n audio_cache progress\n scrape_jobs app_users\n ranking"
}
valkey: Valkey {
shape: cylinder
label: "Valkey :6379\n\n(presign URL cache\nTTL-based, shared)"
}
meilisearch: Meilisearch {
shape: cylinder
label: "Meilisearch :7700\n\nindices:\n books"
}
}
# ─── Application ──────────────────────────────────────────────────────────────
app: Application {
style.fill: "#eef3ff"
caddy: caddy {
shape: rectangle
label: "Caddy :443 / :80\n(reverse proxy\nauto-HTTPS via Let's Encrypt)"
}
backend: backend {
shape: rectangle
label: "Backend API :8080\n(Go — HTTP API server)"
}
runner: runner {
shape: rectangle
label: "Runner :9091\n(Go — background worker\nscraping + TTS jobs\n/metrics endpoint)"
}
ui: ui {
shape: rectangle
label: "SvelteKit UI :3000\n(adapter-node)"
}
}
# ─── Init → Storage deps ──────────────────────────────────────────────────────
init.minio-init -> storage.minio: create buckets {style.stroke-dash: 4}
init.pb-init -> storage.pocketbase: bootstrap schema {style.stroke-dash: 4}
# ─── App → Storage ────────────────────────────────────────────────────────────
app.backend -> storage.minio: blobs (chapters, audio,\navatars, browse)
app.backend -> storage.pocketbase: structured records\n(books, progress, jobs…)
app.backend -> storage.valkey: cache presigned URLs\n(SET/GET with TTL)
app.runner -> storage.minio: write chapter markdown\n& audio MP3s
app.runner -> storage.pocketbase: read/update scrape jobs\nwrite book records
app.runner -> storage.meilisearch: index books on\nscrape completion
app.ui -> storage.valkey: read presigned URL cache\n(replaces in-process Map)
# ─── App internal ─────────────────────────────────────────────────────────────
app.ui -> app.backend: REST API calls\n(server-side)
# ─── Caddy routing ────────────────────────────────────────────────────────────
app.caddy -> app.ui: /* (proxy)
app.caddy -> app.backend: /api/*, /health (proxy)
app.caddy -> storage.minio: /s3/* (proxy, internal only)
# ─── External → App ───────────────────────────────────────────────────────────
app.runner -> novelfire: scrape\n(HTTP GET)
app.runner -> kokoro: TTS generation\n(HTTP POST)
app.caddy -> letsencrypt: ACME certificate\n(TLS-ALPN-01)
# ─── Browser ──────────────────────────────────────────────────────────────────
browser -> app.caddy: HTTPS :443\n(single entry point)

View File

@@ -0,0 +1,59 @@
```mermaid
graph LR
%% ── External ──────────────────────────────────────────────────────────
NF([novelfire.net])
KK([Kokoro-FastAPI TTS])
LE([Let's Encrypt])
CL([Browser / iOS App])
%% ── Init containers ───────────────────────────────────────────────────
subgraph INIT["Init containers (one-shot)"]
MI[minio-init\nmc: create buckets]
PI[pb-init\nbootstrap collections]
end
%% ── Storage ───────────────────────────────────────────────────────────
subgraph STORAGE["Storage"]
MN[(MinIO :9000\nchapters · audio\navatars · browse)]
PB[(PocketBase :8090\nbooks · chapters_idx\naudio_cache · progress\nscrape_jobs · app_users · ranking)]
VK[(Valkey :6379\npresign URL cache\nTTL-based · shared)]
MS[(Meilisearch :7700\nindex: books)]
end
%% ── Application ───────────────────────────────────────────────────────
subgraph APP["Application"]
CD[Caddy :443/:80\nreverse proxy\nauto-HTTPS]
BE[Backend API :8080\nGo HTTP server]
RN[Runner :9091\nGo background worker\n/metrics endpoint]
UI[SvelteKit UI :3000\nadapter-node]
end
%% ── Init → Storage ────────────────────────────────────────────────────
MI -.->|create buckets| MN
PI -.->|bootstrap schema| PB
%% ── App → Storage ─────────────────────────────────────────────────────
BE -->|blobs| MN
BE -->|structured records| PB
BE -->|cache presigned URLs| VK
RN -->|chapter markdown & audio| MN
RN -->|read/update jobs & books| PB
RN -->|index books on scrape| MS
UI -->|read presign cache| VK
%% ── App internal ──────────────────────────────────────────────────────
UI -->|REST API| BE
%% ── Caddy routing ─────────────────────────────────────────────────────
CD -->|/* proxy| UI
CD -->|/api/* /health proxy| BE
CD -->|/s3/* proxy internal| MN
%% ── Runner → External ─────────────────────────────────────────────────
RN -->|scrape HTTP GET| NF
RN -->|TTS HTTP POST| KK
CD -->|ACME certificate| LE
%% ── Client ────────────────────────────────────────────────────────────
CL -->|HTTPS :443 single entry| CD
```

125
v3/docs/architecture.svg Normal file

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 51 KiB

View File

@@ -0,0 +1,102 @@
# Data Flow — Scrape & TTS Job Pipeline
How content moves from novelfire.net through the runner into storage, and how
audio is generated on-demand via the backend.
## Catalogue Scrape Pipeline
The runner performs a background catalogue walk on startup and then on a
configurable interval (`RUNNER_CATALOGUE_REFRESH_INTERVAL`, default 24 h).
```mermaid
flowchart TD
A([Runner starts / refresh tick]) --> B[Walk novelfire.net catalogue\npages 1…N]
B --> C{Book already\nin PocketBase?}
C -- no --> D[Scrape book metadata\ntitle · author · genres\ncover · summary · status]
C -- yes --> E[Check for new chapters\ncompare total_chapters]
D --> F[Write BookMeta\nto PocketBase books]
E --> G{New chapters\nfound?}
G -- no --> Z([Done — next book])
G -- yes --> H
F --> H[Scrape chapter list\n→ chapters_idx in PocketBase]
H --> I[Worker pool — N goroutines\nRUNNER_MAX_CONCURRENT_SCRAPE]
I --> J[For each missing chapter:\nGET chapter HTML from novelfire.net]
J --> K[Parse HTML → Markdown\nhtmlutil.NodeToMarkdown]
K --> L[PUT object to MinIO\nlibnovel-chapters/{slug}/{n}.md]
L --> M[Upsert book doc\nto Meilisearch index: books]
M --> Z
F --> M
```
## On-Demand Single-Book Scrape
Triggered when a user visits `/books/{slug}` and the book is not in PocketBase.
The UI calls `GET /api/book-preview/{slug}` → backend enqueues a task.
```mermaid
sequenceDiagram
actor U as User
participant UI as SvelteKit UI
participant BE as Backend API
participant TQ as Task Queue (PocketBase)
participant RN as Runner
participant NF as novelfire.net
participant PB as PocketBase
participant MN as MinIO
participant MS as Meilisearch
U->>UI: Visit /books/{slug}
UI->>BE: GET /api/book-preview/{slug}
BE->>PB: getBook(slug) — not found
BE->>TQ: INSERT scrape_task (slug, status=pending)
BE-->>UI: 202 {task_id, message}
UI-->>U: "Scraping…" placeholder
RN->>TQ: Poll for pending tasks
TQ-->>RN: scrape_task (slug)
RN->>NF: GET novelfire.net/book/{slug}
NF-->>RN: HTML
RN->>PB: upsert book + chapters_idx
RN->>MN: PUT chapter objects
RN->>MS: UpsertBook doc
RN->>TQ: UPDATE task status=done
U->>UI: Poll GET /api/scrape/tasks/{task_id}
UI->>BE: GET /api/scrape/status
BE->>TQ: get task
TQ-->>BE: status=done
BE-->>UI: {status:"done"}
UI-->>U: Redirect to /books/{slug}
```
## TTS Audio Generation Pipeline
Audio is generated lazily: on first request the job is enqueued; subsequent
requests poll for completion and then stream from MinIO via presigned URL.
```mermaid
flowchart TD
A([POST /api/audio/{slug}/{n}\nbody: voice=af_bella]) --> B{Audio already\nin MinIO?}
B -- yes --> C[200 status: done]
B -- no --> D{Job already\nin queue?}
D -- yes pending/generating --> E[202 task_id + status]
D -- no --> F[INSERT audio_task\nstatus=pending\nin PocketBase]
F --> E
G([Runner polls task queue]) --> H[Claim audio_task\nstatus=generating]
H --> I[GET /api/chapter-text/{slug}/{n}\nfrom backend — plain text]
I --> J[POST /v1/audio/speech\nto Kokoro-FastAPI\nbody: text + voice]
J --> K[Stream MP3 response]
K --> L[PUT object to MinIO\nlibnovel-audio/{slug}/{n}/{voice}.mp3]
L --> M[UPDATE audio_task\nstatus=done]
N([Client polls\nGET /api/audio/status/{slug}/{n}]) --> O{status?}
O -- pending/generating --> N
O -- done --> P[GET /api/presign/audio/{slug}/{n}]
P --> Q{Valkey cache hit?}
Q -- yes --> R[302 → presigned URL]
Q -- no --> S[GeneratePresignedURL\nfrom MinIO — TTL 1h]
S --> T[Cache in Valkey\nTTL 3500s]
T --> R
R --> U([Client streams audio\ndirectly from MinIO])
```

View File

@@ -0,0 +1,87 @@
# Request Flow
Two representative request paths through the stack: a **page load** (SSR) and a
**media playback** (presigned URL → direct MinIO stream).
## SSR Page Load — Browse / Book Detail
```mermaid
sequenceDiagram
actor C as Browser / iOS App
participant CD as Caddy :443
participant UI as SvelteKit UI :3000
participant BE as Backend API :8080
participant MS as Meilisearch :7700
participant PB as PocketBase :8090
participant VK as Valkey :6379
participant MN as MinIO :9000
C->>CD: HTTPS GET /browse
CD->>UI: proxy /*
UI->>BE: GET /api/catalogue?page=1&sort=popular
BE->>MS: search(query, filters, sort)
MS-->>BE: [{slug, title, …}, …]
BE-->>UI: {books[], page, total, has_next}
UI-->>CD: SSR HTML
CD-->>C: 200 HTML
Note over C,UI: Infinite scroll — client fetches next page
C->>CD: HTTPS GET /api/browse-page?page=2
CD->>UI: proxy (SvelteKit API route)
UI->>BE: GET /api/catalogue?page=2
BE->>MS: search(…)
MS-->>BE: next page
BE-->>UI: {books[], …}
UI-->>C: JSON
```
## Audio Playback — Presigned URL Flow
```mermaid
sequenceDiagram
actor C as Browser / iOS App
participant CD as Caddy :443
participant UI as SvelteKit UI :3000
participant BE as Backend API :8080
participant VK as Valkey :6379
participant MN as MinIO :9000
C->>CD: GET /api/presign/audio/{slug}/{n}?voice=af_bella
CD->>BE: proxy /api/*
BE->>VK: GET presign:audio:{slug}:{n}:{voice}
alt cache hit
VK-->>BE: presigned URL (TTL remaining)
BE-->>C: 302 redirect → presigned URL
else cache miss
BE->>MN: GeneratePresignedURL(audio-bucket, key, 1h)
MN-->>BE: presigned URL
BE->>VK: SET presign:audio:… EX 3500
BE-->>C: 302 redirect → presigned URL
end
C->>MN: GET presigned URL (direct, no proxy)
MN-->>C: audio/mpeg stream
```
## Chapter Read — SSR + Content Fetch
```mermaid
sequenceDiagram
actor C as Browser / iOS App
participant CD as Caddy :443
participant UI as SvelteKit UI :3000
participant BE as Backend API :8080
participant PB as PocketBase :8090
participant MN as MinIO :9000
C->>CD: HTTPS GET /books/{slug}/chapters/{n}
CD->>UI: proxy /*
UI->>PB: getBook(slug) + listChapterIdx(slug)
PB-->>UI: book meta + chapter list
UI->>BE: GET /api/chapter-markdown/{slug}/{n}
BE->>MN: GetObject(chapters-bucket, {slug}/{n}.md)
MN-->>BE: markdown text
BE-->>UI: markdown body
Note over UI: marked() → HTML
UI-->>CD: SSR HTML
CD-->>C: 200 HTML
```