a85636d5db
- New Go backend binary (backend + runner) replacing old scraper/ - Rename SCRAPER_API_URL → BACKEND_API_URL in UI env and docker-compose - Rename scraperFetch → backendFetch across all 19 UI server files - Remove SCRAPER_PROXY env var and proxy transport from browser.Config - Add Meilisearch, Valkey, Caddy to docker-compose - Add docs/: api-endpoints.md, request-flow.mermaid.md, data-flow.mermaid.md
103 lines
3.7 KiB
Markdown
103 lines
3.7 KiB
Markdown
# Data Flow — Scrape & TTS Job Pipeline
|
|
|
|
How content moves from novelfire.net through the runner into storage, and how
|
|
audio is generated on-demand via the backend.
|
|
|
|
## Catalogue Scrape Pipeline
|
|
|
|
The runner performs a background catalogue walk on startup and then on a
|
|
configurable interval (`RUNNER_CATALOGUE_REFRESH_INTERVAL`, default 24 h).
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A([Runner starts / refresh tick]) --> B[Walk novelfire.net catalogue\npages 1…N]
|
|
B --> C{Book already\nin PocketBase?}
|
|
C -- no --> D[Scrape book metadata\ntitle · author · genres\ncover · summary · status]
|
|
C -- yes --> E[Check for new chapters\ncompare total_chapters]
|
|
D --> F[Write BookMeta\nto PocketBase books]
|
|
E --> G{New chapters\nfound?}
|
|
G -- no --> Z([Done — next book])
|
|
G -- yes --> H
|
|
F --> H[Scrape chapter list\n→ chapters_idx in PocketBase]
|
|
H --> I[Worker pool — N goroutines\nRUNNER_MAX_CONCURRENT_SCRAPE]
|
|
I --> J[For each missing chapter:\nGET chapter HTML from novelfire.net]
|
|
J --> K[Parse HTML → Markdown\nhtmlutil.NodeToMarkdown]
|
|
K --> L[PUT object to MinIO\nlibnovel-chapters/{slug}/{n}.md]
|
|
L --> M[Upsert book doc\nto Meilisearch index: books]
|
|
M --> Z
|
|
F --> M
|
|
```
|
|
|
|
## On-Demand Single-Book Scrape
|
|
|
|
Triggered when a user visits `/books/{slug}` and the book is not in PocketBase.
|
|
The UI calls `GET /api/book-preview/{slug}` → backend enqueues a task.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
actor U as User
|
|
participant UI as SvelteKit UI
|
|
participant BE as Backend API
|
|
participant TQ as Task Queue (PocketBase)
|
|
participant RN as Runner
|
|
participant NF as novelfire.net
|
|
participant PB as PocketBase
|
|
participant MN as MinIO
|
|
participant MS as Meilisearch
|
|
|
|
U->>UI: Visit /books/{slug}
|
|
UI->>BE: GET /api/book-preview/{slug}
|
|
BE->>PB: getBook(slug) — not found
|
|
BE->>TQ: INSERT scrape_task (slug, status=pending)
|
|
BE-->>UI: 202 {task_id, message}
|
|
UI-->>U: "Scraping…" placeholder
|
|
|
|
RN->>TQ: Poll for pending tasks
|
|
TQ-->>RN: scrape_task (slug)
|
|
RN->>NF: GET novelfire.net/book/{slug}
|
|
NF-->>RN: HTML
|
|
RN->>PB: upsert book + chapters_idx
|
|
RN->>MN: PUT chapter objects
|
|
RN->>MS: UpsertBook doc
|
|
RN->>TQ: UPDATE task status=done
|
|
|
|
U->>UI: Poll GET /api/scrape/tasks/{task_id}
|
|
UI->>BE: GET /api/scrape/status
|
|
BE->>TQ: get task
|
|
TQ-->>BE: status=done
|
|
BE-->>UI: {status:"done"}
|
|
UI-->>U: Redirect to /books/{slug}
|
|
```
|
|
|
|
## TTS Audio Generation Pipeline
|
|
|
|
Audio is generated lazily: on first request the job is enqueued; subsequent
|
|
requests poll for completion and then stream from MinIO via presigned URL.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A([POST /api/audio/{slug}/{n}\nbody: voice=af_bella]) --> B{Audio already\nin MinIO?}
|
|
B -- yes --> C[200 status: done]
|
|
B -- no --> D{Job already\nin queue?}
|
|
D -- yes pending/generating --> E[202 task_id + status]
|
|
D -- no --> F[INSERT audio_task\nstatus=pending\nin PocketBase]
|
|
F --> E
|
|
|
|
G([Runner polls task queue]) --> H[Claim audio_task\nstatus=generating]
|
|
H --> I[GET /api/chapter-text/{slug}/{n}\nfrom backend — plain text]
|
|
I --> J[POST /v1/audio/speech\nto Kokoro-FastAPI\nbody: text + voice]
|
|
J --> K[Stream MP3 response]
|
|
K --> L[PUT object to MinIO\nlibnovel-audio/{slug}/{n}/{voice}.mp3]
|
|
L --> M[UPDATE audio_task\nstatus=done]
|
|
|
|
N([Client polls\nGET /api/audio/status/{slug}/{n}]) --> O{status?}
|
|
O -- pending/generating --> N
|
|
O -- done --> P[GET /api/presign/audio/{slug}/{n}]
|
|
P --> Q{Valkey cache hit?}
|
|
Q -- yes --> R[302 → presigned URL]
|
|
Q -- no --> S[GeneratePresignedURL\nfrom MinIO — TTL 1h]
|
|
S --> T[Cache in Valkey\nTTL 3500s]
|
|
T --> R
|
|
R --> U([Client streams audio\ndirectly from MinIO])
|
|
```
|