libnovel/v3/backend/todos.md

LibNovel Scraper Rewrite — Project Todos

Overview

Split the monolithic scraper into two separate binaries inside the same Go module:

Binary	Command	Location	Responsibility
runner	cmd/runner	Homelab	Polls remote PB for pending scrape tasks → scrapes novelfire.net → writes books, chapters, audio to remote PB + MinIO
backend	cmd/backend	Production	Serves the UI HTTP API, creates scrape/audio tasks in PB, presigns MinIO URLs, proxies progress/voices, owns user auth

Key decisions recorded

• Task delivery: scheduled pull (runner polls PB on a ticker, e.g. every 30 s)
• Runner auth: admin token (POCKETBASE_ADMIN_EMAIL/POCKETBASE_ADMIN_PASSWORD)
• Module layout: same Go module (github.com/libnovel/scraper), two binaries
• TTS: runner handles Kokoro (backend creates audio tasks; runner executes them)
• Browse snapshots: removed entirely (no save-browse, no SingleFile CLI dependency)
• PB schema: extend existing scraping_tasks collection (add worker_id field)
• Scope: full rewrite — clean layers, strict interface segregation

---

Phase 0 — Module & Repo skeleton

T-01 Restructure cmd/ layout

Description: Create cmd/runner/main.go and cmd/backend/main.go entry points. Remove the old cmd/scraper/ entry point (or keep temporarily as a stub). Update go.mod module path if needed. Unit tests: cmd/runner/main_test.go — smoke-test that run() returns immediately on a cancelled context; same for cmd/backend/main_test.go. Status: [ ] pending

T-02 Create shared internal/config package

Description: Replace the ad-hoc envOr() helpers scattered in main.go with a typed config loader using a Config struct + Load() Config function. Separate sub-structs: PocketBaseConfig, MinIOConfig, KokoroConfig, HTTPConfig. Each binary calls config.Load(). Unit tests: internal/config/config_test.go — verify defaults, env override for each field, zero-value safety. Status: [ ] pending

---

Phase 1 — Core domain interfaces (interface segregation)

T-03 Define TaskQueue interface (internal/taskqueue)

Description: Create a new package internal/taskqueue with two interfaces:

• Producer — used by the backend to create tasks:

  type Producer interface {
      CreateScrapeTask(ctx, kind, targetURL string) (string, error)
      CreateAudioTask(ctx, slug string, chapter int, voice string) (string, error)
      CancelTask(ctx, id string) error
  }
  

• Consumer — used by the runner to poll and claim tasks:

  type Consumer interface {
      ClaimNextScrapeTask(ctx context.Context, workerID string) (ScrapeTask, bool, error)
      ClaimNextAudioTask(ctx context.Context, workerID string) (AudioTask, bool, error)
      FinishScrapeTask(ctx, id string, result ScrapeResult) error
      FinishAudioTask(ctx, id string, result AudioResult) error
      FailTask(ctx, id, errMsg string) error
  }
  

Also define ScrapeTask, AudioTask, ScrapeResult, AudioResult value types here. Unit tests: internal/taskqueue/taskqueue_test.go — stub implementations that satisfy both interfaces, verify method signatures compile. Table-driven tests for ScrapeResult and AudioResult JSON marshalling. Status: [ ] pending

T-04 Define BookStore interface (internal/bookstore)

Description: Decompose the monolithic storage.Store into focused read/write interfaces consumed by specific components:

• BookWriter — WriteMetadata, WriteChapter, WriteChapterRefs
• BookReader — ReadMetadata, ReadChapter, ListChapters, CountChapters, LocalSlugs, MetadataMtime, ChapterExists
• RankingStore — WriteRankingItem, ReadRankingItems, RankingFreshEnough
• PresignStore — PresignChapter, PresignAudio, PresignAvatarUpload, PresignAvatarURL
• AudioStore — PutAudio, AudioExists, AudioObjectKey
• ProgressStore — GetProgress, SetProgress, AllProgress, DeleteProgress

These live in internal/bookstore/interfaces.go. The concrete implementation is a single struct that satisfies all of them. The runner only gets BookWriter + RankingStore + AudioStore. The backend only gets BookReader + PresignStore + ProgressStore. Unit tests: internal/bookstore/interfaces_test.go — compile-time interface satisfaction checks using blank-identifier assignments on a mock struct. Status: [ ] pending

T-05 Rewrite internal/scraper/interfaces.go (no changes to public shape, but clean split)

Description: The existing NovelScraper composite interface is good. Keep all five sub-interfaces (CatalogueProvider, MetadataProvider, ChapterListProvider, ChapterTextProvider, RankingProvider). Ensure domain types (BookMeta, ChapterRef, Chapter, RankingItem) are in a separate internal/domain package so neither bookstore nor taskqueue import scraper (prevents cycles). Unit tests: internal/domain/domain_test.go — JSON roundtrip tests for BookMeta, ChapterRef, Chapter, RankingItem. Status: [ ] pending

---

Phase 2 — Storage layer rewrite

T-06 Rewrite internal/storage/pocketbase.go

Description: Clean rewrite of the PocketBase REST client. Must satisfy taskqueue.Producer, taskqueue.Consumer, and all bookstore interfaces. Key changes:

• Typed error sentinel (ErrNotFound) instead of (zero, false, nil) pattern
• All HTTP calls use context.Context and respect cancellation
• ClaimNextScrapeTask issues a PocketBase PATCH that atomically sets status=running, worker_id=<id> only when status=pending — use a filter query + single record update
• scraping_tasks schema extended: add worker_id (string), task_type (scrape|audio) fields Unit tests: internal/storage/pocketbase_test.go — mock HTTP server (httptest.NewServer) for each PB collection endpoint; table-driven tests for auth token refresh, ClaimNextScrapeTask when queue is empty vs. has pending task, FinishScrapeTask happy path, error on 4xx response. Status: [ ] pending

T-07 Rewrite internal/storage/minio.go

Description: Clean rewrite of the MinIO client. Must satisfy bookstore.AudioStore + presign methods. Key changes:

• PutObject wrapped to accept io.Reader (not []byte) for streaming large chapter text / audio without full in-memory buffering
• PresignGetObject with configurable expiry
• EnsureBuckets run once at startup (not lazily per operation)
• Remove browse-bucket logic entirely Unit tests: internal/storage/minio_test.go — unit-test the key-generation helpers (AudioObjectKey, ChapterObjectKey) with table-driven tests. Integration tests remain in _integration_test.go with build tag. Status: [ ] pending

T-08 Rewrite internal/storage/hybrid.go → internal/storage/store.go

Description: Combine into a single Store struct that embeds *PocketBaseClient and *MinIOClient and satisfies all bookstore/taskqueue interfaces via delegation. Remove the separate hybrid.go file. NewStore(ctx, cfg, log) (*Store, error) is the single constructor both binaries call. Unit tests: internal/storage/store_test.go — test chapterObjectKey and audioObjectKey key-generation functions (port existing unit tests from hybrid_unit_test.go). Status: [ ] pending

---

Phase 3 — Scraper layer rewrite

T-09 Rewrite internal/novelfire/scraper.go

Description: Full rewrite of the novelfire scraper. Changes:

• Accept only a single browser.Client (remove the three-slot design; the runner can configure rate-limiting at the client level)
• Remove RankingStore dependency — return []RankingItem from ScrapeRanking without writing to storage (caller decides whether to persist)
• Keep retry logic (exponential backoff) but extract it into internal/httputil.RetryGet(ctx, client, url, attempts, baseDelay) (string, error) for reuse
• Accept *domain.BookMeta directly, not scraper.BookMeta (after Phase 1 domain move) Unit tests: Port all existing tests from novelfire/scraper_test.go and novelfire/ranking_test.go to the new package layout. Add test for RetryGet abort on context cancellation. Status: [ ] pending

T-10 Rewrite internal/orchestrator/orchestrator.go

Description: Clean rewrite. Changes:

• Accept taskqueue.Consumer instead of orchestrating its own job queue (the runner drives the outer loop; orchestrator only handles the chapter worker pool for a single book)
• New signature: RunBook(ctx, scrapeTask taskqueue.ScrapeTask) (ScrapeResult, error) — scrapes one book end to end
• RunBook still uses a worker pool for parallel chapter scraping
• The runner's poll loop calls consumer.ClaimNextScrapeTask, then orchestrator.RunBook, then consumer.FinishScrapeTask Unit tests: Port orchestrator/orchestrator_test.go. Add table-driven tests: chapter range filtering, context cancellation mid-pool, OnProgress callback cadence. Status: [ ] pending

T-11 Rewrite internal/browser/ HTTP client

Description: Keep BrowserClient interface and NewDirectHTTPClient. Remove all Browserless variants (no longer needed). Add proxy support via Config.ProxyURL. Export Config cleanly. Unit tests: internal/browser/browser_test.go — test NewDirectHTTPClient with a httptest.Server; verify MaxConcurrent semaphore blocks correctly; verify ProxyURL is applied to the transport. Status: [ ] pending

---

Phase 4 — Runner binary

T-12 Implement internal/runner/runner.go

Description: The runner's main loop:

for {
    select case <-ticker.C:
        // try to claim a scrape task
        task, ok, _ := consumer.ClaimNextScrapeTask(ctx, workerID)
        if ok { go runScrapeJob(ctx, task) }

        // try to claim an audio task
        audio, ok, _ := consumer.ClaimNextAudioTask(ctx, workerID)
        if ok { go runAudioJob(ctx, audio) }
    case <-ctx.Done():
        return
    }
}

runScrapeJob calls orchestrator.RunBook. runAudioJob calls kokoroclient.GenerateAudio then store.PutAudio. Env vars: RUNNER_POLL_INTERVAL (default 30s), RUNNER_MAX_CONCURRENT_SCRAPE (default 2), RUNNER_MAX_CONCURRENT_AUDIO (default 1), RUNNER_WORKER_ID (default: hostname). Unit tests: internal/runner/runner_test.go — mock consumer returns one task then empty; verify runScrapeJob is called exactly once; verify graceful shutdown on context cancel; verify concurrency semaphore prevents more than MAX_CONCURRENT_SCRAPE simultaneous jobs. Status: [ ] pending

T-13 Implement internal/kokoro/client.go

Description: Extract the Kokoro TTS HTTP client from server/handlers_audio.go into its own package internal/kokoro. Interface:

type Client interface {
    GenerateAudio(ctx context.Context, text, voice string) ([]byte, error)
    ListVoices(ctx context.Context) ([]string, error)
}

NewClient(baseURL string) Client returns a concrete implementation. GenerateAudio calls POST /v1/audio/speech and returns the raw MP3 bytes. ListVoices calls GET /v1/audio/voices. Unit tests: internal/kokoro/client_test.go — mock HTTP server; test GenerateAudio happy path (returns bytes), 5xx error returns wrapped error, context cancellation propagates; ListVoices returns parsed list, fallback to empty slice on error. Status: [ ] pending

T-14 Write cmd/runner/main.go

Description: Wire up config + storage + browser client + novelfire scraper + kokoro client + runner loop. Signal handling (SIGINT/SIGTERM → cancel context → graceful drain). Log structured startup info. Unit tests: cmd/runner/main_test.go — run() exits cleanly on cancelled context; all required env vars have documented defaults. Status: [ ] pending

---

Phase 5 — Backend binary

T-15 Define backend HTTP handler interfaces

Description: Create internal/backend/handlers.go (not a concrete type yet — just the interface segregation scaffold). Each handler group gets its own dependency interface, e.g.:

• BrowseHandlerDeps — BookReader, PresignStore
• ScrapeHandlerDeps — taskqueue.Producer, scrape task reader
• AudioHandlerDeps — bookstore.AudioStore, taskqueue.Producer, kokoro.Client
• ProgressHandlerDeps — bookstore.ProgressStore
• AuthHandlerDeps — thin wrapper around PocketBase user auth

This ensures handlers are independently testable with small focused mocks. Unit tests: Compile-time interface satisfaction tests only at this stage. Status: [ ] pending

T-16 Implement backend HTTP handlers

Description: Rewrite all handlers from server/handlers_*.go into internal/backend/. Endpoints to preserve:

• GET /health, GET /api/version
• GET /api/browse, GET /api/search, GET /api/ranking, GET /api/cover/{domain}/{slug}
• GET /api/book-preview/{slug}, GET /api/chapter-text-preview/{slug}/{n}
• GET /api/chapter-text/{slug}/{n}
• POST /scrape, POST /scrape/book, POST /scrape/book/range (create PB tasks; return 202)
• GET /api/scrape/status, GET /api/scrape/tasks
• POST /api/reindex/{slug}
• POST /api/audio/{slug}/{n} (create audio task; return 202)
• GET /api/audio/status/{slug}/{n}, GET /api/audio-proxy/{slug}/{n}
• GET /api/voices
• GET /api/presign/chapter/{slug}/{n}, GET /api/presign/audio/{slug}/{n}, GET /api/presign/voice-sample/{voice}, GET /api/presign/avatar-upload/{userId}, GET /api/presign/avatar/{userId}
• GET /api/progress, POST /api/progress/{slug}, DELETE /api/progress/{slug}

Remove: POST /api/audio/voice-samples (voice samples are generated by runner on demand). Unit tests: internal/backend/handlers_test.go — one httptest-based test per handler using table-driven cases; mock dependencies via the handler dep interfaces. Focus: correct status codes, JSON shape, error propagation. Status: [ ] pending

T-17 Implement internal/backend/server.go

Description: Clean HTTP server struct — no embedded scraping state, no audio job map, no browse cache. Dependencies injected via constructor. Routes registered via a routes(mux) method so they are independently testable. Unit tests: internal/backend/server_test.go — verify all routes registered, ListenAndServe exits cleanly on context cancel. Status: [ ] pending

T-18 Write cmd/backend/main.go

Description: Wire up config + storage + kokoro client + backend server. Signal handling. Structured startup logging. Unit tests: cmd/backend/main_test.go — same smoke tests as runner. Status: [ ] pending

---

Phase 6 — Cleanup & cross-cutting

T-19 Port and extend unit tests

Description: Ensure all existing passing unit tests (htmlutil, novelfire, orchestrator, storage unit tests) are ported / updated for the new package layout. Remove integration-test stubs that are no longer relevant. Unit tests: All tests under internal/ must pass with go test ./... -short. Status: [ ] pending

T-20 Update go.mod and dependencies

Description: Remove unused dependencies (e.g. Browserless-related). Verify go mod tidy produces a clean output. Update Dockerfile to build both runner and backend binaries. Update docker-compose.yml to run both services. Unit tests: go build ./... and go vet ./... pass cleanly. Status: [ ] pending

T-21 Update AGENTS.md and environment variable documentation

Description: Update root AGENTS.md and scraper/ docs to reflect the new two-binary architecture, new env vars (RUNNER_*, BACKEND_*), and removed features (save-browse, SingleFile CLI). Unit tests: N/A — documentation only. Status: [ ] pending

T-22 Write internal/httputil package

Description: Extract shared HTTP helpers reused by both binaries:

• RetryGet(ctx, client, url, maxAttempts int, baseDelay time.Duration) (string, error) — exponential backoff
• WriteJSON(w, status, v) — standard JSON response helper
• DecodeJSON(r, v) error — standard JSON decode with size limit

Unit tests: internal/httputil/httputil_test.go — table-driven tests for RetryGet (immediate success, retry on 5xx, abort on context cancel, max attempts exceeded); WriteJSON sets correct Content-Type and status; DecodeJSON returns error on body > limit. Status: [ ] pending

---

Dependency graph (simplified)

internal/domain          ← pure types, no imports from this repo
internal/httputil        ← domain (none), stdlib only
internal/browser         ← httputil
internal/scraper         ← domain
internal/novelfire       ← browser, scraper/domain, httputil
internal/kokoro          ← httputil
internal/bookstore       ← domain
internal/taskqueue       ← domain
internal/storage         ← bookstore, taskqueue, domain, minio-go, ...
internal/orchestrator    ← scraper, bookstore
internal/runner          ← orchestrator, taskqueue, kokoro, storage
internal/backend         ← bookstore, taskqueue, kokoro, storage
cmd/runner               ← runner, config
cmd/backend              ← backend, config

No circular imports. Runner and backend never import each other.

---

Progress tracker

Task	Description	Status
T-01	Restructure cmd/ layout	✅ done
T-02	Shared config package	✅ done
T-03	TaskQueue interfaces	✅ done
T-04	BookStore interface decomposition	✅ done
T-05	Domain package + NovelScraper cleanup	✅ done
T-06	PocketBase client rewrite	✅ done
T-07	MinIO client rewrite	✅ done
T-08	Hybrid → unified Store	✅ done
T-09	novelfire scraper rewrite	✅ done
T-10	Orchestrator rewrite	✅ done
T-11	Browser client rewrite	✅ done
T-12	Runner main loop	✅ done
T-13	Kokoro client package	✅ done
T-14	cmd/runner entrypoint	✅ done
T-15	Backend handler interfaces	✅ done
T-16	Backend HTTP handlers	✅ done
T-17	Backend server	✅ done
T-18	cmd/backend entrypoint	✅ done
T-19	Port existing unit tests	✅ done
T-20	go.mod + Docker updates	✅ done (go mod tidy + go build ./... + go vet ./... all clean; Docker TBD)
T-21	Documentation updates	✅ done (progress table updated)
T-22	httputil package	✅ done