libnovel/docs/mermaid/data-flow.mermaid.md

Data Flow — Scrape & TTS Job Pipeline

How content moves from novelfire.net through the runner into storage, and how audio is generated on-demand via the backend.

Catalogue Scrape Pipeline

The runner performs a background catalogue walk on startup and then on a configurable interval (RUNNER_CATALOGUE_REFRESH_INTERVAL, default 24 h).

flowchart TD
    A([Runner starts / refresh tick]) --> B[Walk novelfire.net catalogue\npages 1…N]
    B --> C{Book already\nin PocketBase?}
    C -- no --> D[Scrape book metadata\ntitle · author · genres\ncover · summary · status]
    C -- yes --> E[Check for new chapters\ncompare total_chapters]
    D --> F[Write BookMeta\nto PocketBase books]
    E --> G{New chapters\nfound?}
    G -- no --> Z([Done — next book])
    G -- yes --> H
    F --> H[Scrape chapter list with upTo limit\n→ chapters_idx in PocketBase\nretries on 429 with Retry-After backoff]
    H --> I[Worker pool — N goroutines\nRUNNER_MAX_CONCURRENT_SCRAPE]
    I --> J[For each missing chapter:\nGET chapter HTML from novelfire.net]
    J --> K[Parse HTML → Markdown\nhtmlutil.NodeToMarkdown]
    K --> L[PUT object to MinIO\nchapters/{slug}/{n}.md]
    L --> M[Upsert book doc\nto Meilisearch index: books]
    M --> Z
    F --> M

On-Demand Single-Book Scrape

Triggered when a user visits /books/{slug} and the book is not in PocketBase. The UI calls GET /api/book-preview/{slug} → backend enqueues a scrape task.

sequenceDiagram
    actor U as User
    participant UI as SvelteKit UI
    participant BE as Backend API
    participant TQ as Task Queue (PocketBase)
    participant RN as Runner
    participant NF as novelfire.net
    participant PB as PocketBase
    participant MN as MinIO
    participant MS as Meilisearch

    U->>UI: Visit /books/{slug}
    UI->>BE: GET /api/book-preview/{slug}
    BE->>PB: getBook(slug) — not found
    BE->>TQ: INSERT scrape_task (slug, status=pending)
    BE-->>UI: 202 {task_id, message}
    UI-->>U: "Scraping…" placeholder

    RN->>TQ: Poll for pending tasks
    TQ-->>RN: scrape_task (slug)
    RN->>NF: GET novelfire.net/book/{slug}
    NF-->>RN: HTML
    RN->>PB: upsert book + chapters_idx
    RN->>MN: PUT chapter objects
    RN->>MS: UpsertBook doc
    RN->>TQ: UPDATE task status=done

    U->>UI: Poll GET /api/scrape/tasks/{task_id}
    UI->>BE: GET /api/scrape/status
    BE->>TQ: get task
    TQ-->>BE: status=done
    BE-->>UI: {status:"done"}
    UI-->>U: Redirect to /books/{slug}

TTS Audio Generation Pipeline

Audio is generated lazily: on first request the job is enqueued; subsequent requests poll for completion and then stream from MinIO via presigned URL.

flowchart TD
    A(["POST /api/audio/{slug}/{n}\nbody: voice=af_bella"]) --> B{Audio already\nin MinIO?}
    B -- yes --> C[200 status: done]
    B -- no --> D{Job already\nin queue?}
    D -- "yes pending/generating" --> E[202 task_id + status]
    D -- no --> F[INSERT audio_task\nstatus=pending\nin PocketBase]
    F --> E

    G([Runner polls task queue]) --> H[Claim audio_task\nstatus=generating]
    H --> I["GET /api/chapter-text/{slug}/{n}\nfrom backend — plain text"]
    I --> J[POST /v1/audio/speech\nto Kokoro-FastAPI\nbody: text + voice]
    J --> K[Stream MP3 response]
    K --> L[PUT object to MinIO\naudio/{slug}/{n}/{voice}.mp3]
    L --> M[UPDATE audio_task\nstatus=done]

    N(["Client polls\nGET /api/audio/status/{slug}/{n}"]) --> O{status?}
    O -- "pending/generating" --> N
    O -- done --> P["GET /api/presign/audio/{slug}/{n}"]
    P --> Q{Valkey cache hit?}
    Q -- yes --> R[302 → presigned URL]
    Q -- no --> S[GeneratePresignedURL\nfrom MinIO — TTL 1h]
    S --> T[Cache in Valkey\nTTL 3500s]
    T --> R
    R --> U([Client streams audio\ndirectly from MinIO])