Compare commits

...

9 Commits

Author SHA1 Message Date
Admin
428b57732e fix(ui): resolve avatar URL from MinIO; fall back to OAuth provider URL
Some checks failed
CI / Backend (push) Successful in 50s
CI / UI (push) Successful in 34s
Release / Test backend (push) Successful in 37s
Release / Check ui (push) Successful in 45s
Release / Docker / caddy (push) Successful in 35s
CI / Backend (pull_request) Successful in 49s
CI / UI (pull_request) Successful in 40s
Release / Docker / backend (push) Failing after 52s
Release / Docker / runner (push) Failing after 58s
Release / Docker / ui (push) Failing after 55s
Release / Gitea Release (push) Has been skipped
Add resolveAvatarUrl(userId, storedValue) helper that tries MinIO first,
then falls back to the stored HTTP URL for OAuth users (Google/GitHub)
who have never uploaded a custom avatar.

Add getUserById() to pocketbase helpers for batch avatar resolution in
comments. Update all 6 call sites to use the new helper.
2026-03-28 19:23:30 +05:00
Admin
61e77e3e28 ci: remove Docker builds from CI; keep vet/build/test/type-check only
All checks were successful
CI / Backend (push) Successful in 43s
CI / UI (push) Successful in 37s
CI / UI (pull_request) Successful in 35s
CI / Backend (pull_request) Successful in 1m9s
Docker image builds belong to the release workflow (tag-triggered).
CI now runs: go vet, go build (backend/runner/healthcheck), go test,
svelte-check, and UI vite build — fast feedback without Docker overhead.
Also triggers on all branches, not just main/master.
2026-03-28 19:08:17 +05:00
Admin
b363c151a5 fix(ui): fix catalogue filters in Svelte 5; improve build badge visibility
Some checks failed
Release / Test backend (push) Successful in 40s
Release / Check ui (push) Successful in 49s
Release / Docker / caddy (push) Failing after 1m0s
CI / Test backend (pull_request) Successful in 40s
CI / Docker / caddy (pull_request) Failing after 21s
CI / Check ui (pull_request) Successful in 48s
Release / Docker / ui (push) Successful in 2m49s
Release / Docker / runner (push) Successful in 3m10s
Release / Docker / backend (push) Successful in 3m47s
Release / Gitea Release (push) Has been skipped
CI / Docker / ui (pull_request) Successful in 1m31s
CI / Docker / backend (pull_request) Successful in 3m13s
CI / Docker / runner (pull_request) Successful in 3m59s
- Catalogue filter selects: replace value= initializer with bind:value +
  onchange goto() so filters navigate immediately on change (no Apply button)
- Add selected= on each <option> for correct DOM initialisation in Svelte 5
- Build badge: give distinct bg-zinc-800 pill, visible zinc-300/400/500 text
  instead of zinc-700/zinc-800 which blended into the footer background
2026-03-28 19:01:58 +05:00
Admin
aef9e04419 fix(runner): harden catalogue scrape against 429s; disable sourcemap upload
Some checks failed
Release / Test backend (push) Successful in 29s
Release / Check ui (push) Successful in 43s
CI / Test backend (pull_request) Successful in 28s
Release / Docker / caddy (push) Successful in 1m8s
CI / Docker / caddy (pull_request) Failing after 39s
CI / Check ui (pull_request) Successful in 55s
Release / Docker / runner (push) Successful in 2m22s
Release / Docker / ui (push) Successful in 2m20s
CI / Docker / backend (pull_request) Successful in 2m32s
CI / Docker / ui (pull_request) Successful in 1m28s
CI / Docker / runner (pull_request) Successful in 2m25s
Release / Docker / backend (push) Successful in 2m33s
Release / Gitea Release (push) Failing after 2s
- scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) +
  500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata
  also switched to retryGet so a single 429 on a book page is retried rather
  than aborting the whole refresh.
- catalogue_refresh.go: per-book delay is now configurable
  (RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter
  applied before every metadata fetch. Only metadata is scraped here —
  chapters are fetched on-demand, not during catalogue refresh. Progress
  logged every 50 books instead of 100.
- config.go / runner.go / main.go: add CatalogueRequestDelay field wired
  from RUNNER_CATALOGUE_REQUEST_DELAY env var.
- release.yaml: comment out upload-sourcemaps job and remove it from the
  release needs; GlitchTip auth token needs refreshing after DB wipe.
2026-03-28 16:28:36 +05:00
Admin
58e78cd34d fix(infra): make MEILI_URL configurable; rename Discover → Catalogue in nav
Some checks failed
Release / Test backend (push) Failing after 11s
Release / Docker / caddy (push) Failing after 11s
Release / Check ui (push) Failing after 11s
Release / Docker / backend (push) Has been skipped
Release / Docker / runner (push) Has been skipped
Release / Upload source maps (push) Has been skipped
Release / Docker / ui (push) Has been skipped
Release / Gitea Release (push) Has been skipped
CI / Check ui (pull_request) Successful in 49s
CI / Docker / ui (pull_request) Successful in 1m37s
CI / Docker / caddy (pull_request) Successful in 4m4s
CI / Test backend (pull_request) Successful in 4m29s
CI / Docker / runner (pull_request) Failing after 1m27s
CI / Docker / backend (pull_request) Successful in 1m41s
- docker-compose.yml: MEILI_URL in x-infra-env was hardcoded to the local
  meilisearch container, ignoring the Doppler MEILI_URL secret entirely.
  Changed to "${MEILI_URL:-http://meilisearch:7700}" so prod reads from
  https://search.libnovel.cc while local dev keeps the container default.
- ui/+layout.svelte: rename nav + footer label from "Discover" to "Catalogue"
  (desktop nav, mobile menu, footer — all three occurrences).
2026-03-28 16:15:04 +05:00
Admin
c5c167035d fix(tts): fix pocket-tts voices missing in UI and 500 on first TTS enqueue
Some checks failed
CI / Test backend (pull_request) Successful in 40s
CI / Check ui (pull_request) Successful in 45s
Release / Test backend (push) Successful in 40s
Release / Check ui (push) Successful in 41s
Release / Docker / caddy (push) Successful in 1m8s
CI / Docker / runner (pull_request) Failing after 35s
CI / Docker / caddy (pull_request) Successful in 3m39s
CI / Docker / ui (pull_request) Successful in 1m12s
CI / Docker / backend (pull_request) Successful in 3m24s
Release / Upload source maps (push) Failing after 43s
Release / Docker / runner (push) Successful in 2m49s
Release / Docker / ui (push) Successful in 2m51s
Release / Docker / backend (push) Failing after 7m14s
Release / Gitea Release (push) Has been skipped
- Add POCKET_TTS_URL env to backend service in docker-compose.yml so
  pocket-tts voices appear in the voice selector (Doppler secret existed
  but the env var was never passed to the container)
- Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of
  single-quoted string, causing the duplicate-task guard to always miss
- Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues
  TTS internally on 404; AudioPlayer now skips the redundant POST and
  polls directly, eliminating the 500 from the PB unique-key conflict
2026-03-28 16:02:46 +05:00
Admin
4a00d953bb feat(ui): show build version in footer; enable watchtower for caddy
Some checks failed
CI / Check ui (pull_request) Successful in 50s
CI / Docker / ui (pull_request) Failing after 38s
CI / Test backend (pull_request) Successful in 3m31s
CI / Docker / backend (pull_request) Failing after 11s
CI / Docker / runner (pull_request) Failing after 11s
CI / Docker / caddy (pull_request) Successful in 11m0s
- Footer now displays build version tag + short commit SHA (from
  PUBLIC_BUILD_VERSION / PUBLIC_BUILD_COMMIT env vars baked in at
  image build time); falls back to 'dev' in local builds
- docker-compose.yml: add watchtower label to caddy service so it
  auto-updates alongside backend/ui/runner on new image pushes
- homelab/docker-compose.yml: use locally-built kokoro-fastapi:latest
  image (consistent with actual homelab setup)
2026-03-28 15:42:22 +05:00
Admin
fe1a933fd0 feat(queue): replace PocketBase polling with Asynq + Redis
Some checks failed
CI / Docker / caddy (pull_request) Failing after 50s
CI / Check ui (pull_request) Successful in 1m5s
Release / Check ui (push) Successful in 45s
Release / Test backend (push) Successful in 1m29s
CI / Docker / ui (pull_request) Successful in 1m28s
Release / Docker / backend (push) Successful in 3m14s
CI / Test backend (pull_request) Failing after 42s
CI / Docker / backend (pull_request) Has been skipped
CI / Docker / runner (pull_request) Has been skipped
Release / Docker / caddy (push) Successful in 6m48s
Release / Docker / ui (push) Successful in 2m8s
Release / Docker / runner (push) Successful in 2m51s
Release / Upload source maps (push) Failing after 53s
Release / Gitea Release (push) Has been skipped
Introduce a Redis-backed Asynq task queue so the runner consumes TTS
jobs pushed by the backend instead of polling PocketBase.

- backend/internal/asynqqueue: Producer and Consumer wrappers
- backend/internal/runner: AsynqRunner mux, per-instance Prometheus
  registry (fixes duplicate-collector panic in tests), redisConnOpt
- backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars
- backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall
  back to legacy poll mode when unset
- Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab
- caddy/Dockerfile: add --with github.com/mholt/caddy-l4
- docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env
- homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on
- homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner)
  and alerting rules / contact-points provisioning
2026-03-28 14:32:40 +05:00
Admin
98e4a87432 feat(tts): dual-engine voice list (kokoro + pocket-tts)
Expose all available voices from both TTS engines via the /api/voices
endpoint. AudioPlayer and profile voice-selector now group voices by
engine and show a labelled optgroup. Voice type carries an engine field
so the chapter-reader can route synthesis to the correct backend.
2026-03-28 14:32:06 +05:00
46 changed files with 2474 additions and 398 deletions

View File

@@ -2,20 +2,14 @@ name: CI
on:
push:
branches: ["main", "master"]
paths:
- "backend/**"
- "ui/**"
- "caddy/**"
- "docker-compose.yml"
- ".gitea/workflows/ci.yaml"
pull_request:
branches: ["main", "master"]
paths:
- "backend/**"
- "ui/**"
- "caddy/**"
- "docker-compose.yml"
- ".gitea/workflows/ci.yaml"
concurrency:
@@ -23,10 +17,13 @@ concurrency:
cancel-in-progress: true
jobs:
# ── backend: vet & test ───────────────────────────────────────────────────────
test-backend:
name: Test backend
# ── Go: vet + build + test ────────────────────────────────────────────────
backend:
name: Backend
runs-on: ubuntu-latest
defaults:
run:
working-directory: backend
steps:
- uses: actions/checkout@v4
@@ -36,16 +33,23 @@ jobs:
cache-dependency-path: backend/go.sum
- name: go vet
working-directory: backend
run: go vet ./...
- name: Build backend
run: go build -o /dev/null ./cmd/backend
- name: Build runner
run: go build -o /dev/null ./cmd/runner
- name: Build healthcheck
run: go build -o /dev/null ./cmd/healthcheck
- name: Run tests
working-directory: backend
run: go test -short -race -count=1 -timeout=60s ./...
# ── ui: type-check & build ────────────────────────────────────────────────────
check-ui:
name: Check ui
# ── UI: type-check + build ────────────────────────────────────────────────
ui:
name: UI
runs-on: ubuntu-latest
defaults:
run:
@@ -67,57 +71,3 @@ jobs:
- name: Build
run: npm run build
# ── docker: validate Dockerfiles build (no push) ──────────────────────────────
docker-backend:
name: Docker / backend
runs-on: ubuntu-latest
needs: [test-backend]
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build
uses: docker/build-push-action@v6
with:
context: backend
target: backend
push: false
docker-runner:
name: Docker / runner
runs-on: ubuntu-latest
needs: [test-backend]
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build
uses: docker/build-push-action@v6
with:
context: backend
target: runner
push: false
docker-ui:
name: Docker / ui
runs-on: ubuntu-latest
needs: [check-ui]
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build
uses: docker/build-push-action@v6
with:
context: ui
push: false
docker-caddy:
name: Docker / caddy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build
uses: docker/build-push-action@v6
with:
context: caddy
push: false

View File

@@ -136,52 +136,51 @@ jobs:
cache-to: type=inline
# ── ui: source map upload ─────────────────────────────────────────────────────
# Builds the UI with source maps and uploads them to GlitchTip so that error
# stack traces resolve to original .svelte/.ts file names and line numbers.
# Runs in parallel with docker-ui (both need check-ui to pass first).
upload-sourcemaps:
name: Upload source maps
runs-on: ubuntu-latest
needs: [check-ui]
defaults:
run:
working-directory: ui
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: ui/package-lock.json
- name: Install dependencies
run: npm ci
- name: Build with source maps
run: npm run build
- name: Download glitchtip-cli
run: |
curl -L "https://gitlab.com/glitchtip/glitchtip-cli/-/jobs/artifacts/v0.1.0/raw/artifacts/glitchtip-cli-linux-x86_64?job=build-linux-x86_64" \
-o /usr/local/bin/glitchtip-cli
chmod +x /usr/local/bin/glitchtip-cli
- name: Inject debug IDs into build artifacts
run: glitchtip-cli sourcemaps inject ./build
env:
SENTRY_URL: https://errors.libnovel.cc/
SENTRY_AUTH_TOKEN: ${{ secrets.GLITCHTIP_AUTH_TOKEN }}
SENTRY_ORG: libnovel
SENTRY_PROJECT: libnovel-ui
- name: Upload source maps to GlitchTip
run: glitchtip-cli sourcemaps upload ./build --release ${{ gitea.ref_name }}
env:
SENTRY_URL: https://errors.libnovel.cc/
SENTRY_AUTH_TOKEN: ${{ secrets.GLITCHTIP_AUTH_TOKEN }}
SENTRY_ORG: libnovel
SENTRY_PROJECT: libnovel-ui
# Commented out: GlitchTip project/auth token needs to be recreated after
# the GlitchTip DB wipe. Re-enable once GLITCHTIP_AUTH_TOKEN is updated.
# upload-sourcemaps:
# name: Upload source maps
# runs-on: ubuntu-latest
# needs: [check-ui]
# defaults:
# run:
# working-directory: ui
# steps:
# - uses: actions/checkout@v4
#
# - uses: actions/setup-node@v4
# with:
# node-version: "22"
# cache: npm
# cache-dependency-path: ui/package-lock.json
#
# - name: Install dependencies
# run: npm ci
#
# - name: Build with source maps
# run: npm run build
#
# - name: Download glitchtip-cli
# run: |
# curl -L "https://gitlab.com/glitchtip/glitchtip-cli/-/jobs/artifacts/v0.1.0/raw/artifacts/glitchtip-cli-linux-x86_64?job=build-linux-x86_64" \
# -o /usr/local/bin/glitchtip-cli
# chmod +x /usr/local/bin/glitchtip-cli
#
# - name: Inject debug IDs into build artifacts
# run: glitchtip-cli sourcemaps inject ./build
# env:
# SENTRY_URL: https://errors.libnovel.cc/
# SENTRY_AUTH_TOKEN: ${{ secrets.GLITCHTIP_AUTH_TOKEN }}
# SENTRY_ORG: libnovel
# SENTRY_PROJECT: libnovel-ui
#
# - name: Upload source maps to GlitchTip
# run: glitchtip-cli sourcemaps upload ./build --release ${{ gitea.ref_name }}
# env:
# SENTRY_URL: https://errors.libnovel.cc/
# SENTRY_AUTH_TOKEN: ${{ secrets.GLITCHTIP_AUTH_TOKEN }}
# SENTRY_ORG: libnovel
# SENTRY_PROJECT: libnovel-ui
# ── docker: ui ────────────────────────────────────────────────────────────────
docker-ui:
@@ -261,7 +260,7 @@ jobs:
release:
name: Gitea Release
runs-on: ubuntu-latest
needs: [docker-backend, docker-runner, docker-ui, docker-caddy, upload-sourcemaps]
needs: [docker-backend, docker-runner, docker-ui, docker-caddy]
steps:
- uses: actions/checkout@v4
with:

View File

@@ -234,3 +234,27 @@ search.libnovel.cc {
reverse_proxy meilisearch:7700
}
# ── Redis TCP proxy: exposes homelab Redis over TLS for Asynq ─────────────────
# The backend (prod) connects to rediss://redis.libnovel.cc:6380 to enqueue
# Asynq jobs. Caddy terminates TLS (Let's Encrypt cert for redis.libnovel.cc)
# and proxies the raw TCP stream to the homelab Redis via this reverse proxy.
#
# NOTE: Redis is NOT running on the prod server — it runs on the homelab
# (192.168.0.109:6379) and is exposed to the internet via this Caddy proxy.
# The homelab Redis is protected by REDIS_PASSWORD (requirepass).
#
# Caddy layer4 app handles this; requires the caddy-l4 module in the build.
{
layer4 {
redis.libnovel.cc:6380 {
route {
tls
proxy {
# Homelab Redis — replace with actual homelab IP or FQDN
upstream {$HOMELAB_REDIS_ADDR:192.168.0.109:6379}
}
}
}
}
}
}

View File

@@ -22,12 +22,16 @@ import (
"time"
"github.com/getsentry/sentry-go"
"github.com/hibiken/asynq"
"github.com/libnovel/backend/internal/asynqqueue"
"github.com/libnovel/backend/internal/backend"
"github.com/libnovel/backend/internal/config"
"github.com/libnovel/backend/internal/kokoro"
"github.com/libnovel/backend/internal/meili"
"github.com/libnovel/backend/internal/otelsetup"
"github.com/libnovel/backend/internal/pockettts"
"github.com/libnovel/backend/internal/storage"
"github.com/libnovel/backend/internal/taskqueue"
)
// version and commit are set at build time via -ldflags.
@@ -100,6 +104,15 @@ func run() error {
kokoroClient = &noopKokoro{}
}
// ── Pocket-TTS (voice list + sample generation; audio generation is the runner's job) ──
var pocketTTSClient pockettts.Client
if cfg.PocketTTS.URL != "" {
pocketTTSClient = pockettts.New(cfg.PocketTTS.URL)
log.Info("pocket-tts voices enabled", "url", cfg.PocketTTS.URL)
} else {
log.Info("POCKET_TTS_URL not set — pocket-tts voices unavailable in backend")
}
// ── Meilisearch (search reads only; indexing is the runner's job) ────────
var searchIndex meili.Client
if cfg.Meilisearch.URL != "" {
@@ -110,6 +123,24 @@ func run() error {
searchIndex = meili.NoopClient{}
}
// ── Task Producer ────────────────────────────────────────────────────────
// When REDIS_ADDR is set the backend dual-writes: PocketBase record (audit)
// + Asynq job (immediate delivery). Otherwise it writes to PocketBase only
// and the runner picks up on the next poll tick.
var producer taskqueue.Producer = store
if cfg.Redis.Addr != "" {
redisOpt, parseErr := parseRedisOpt(cfg.Redis)
if parseErr != nil {
return fmt.Errorf("parse REDIS_ADDR: %w", parseErr)
}
asynqProducer := asynqqueue.NewProducer(store, redisOpt)
defer asynqProducer.Close() //nolint:errcheck
producer = asynqProducer
log.Info("backend: asynq task dispatch enabled", "addr", cfg.Redis.Addr)
} else {
log.Info("backend: poll-mode task dispatch (REDIS_ADDR not set)")
}
// ── Backend server ───────────────────────────────────────────────────────
srv := backend.New(
backend.Config{
@@ -125,10 +156,11 @@ func run() error {
PresignStore: store,
ProgressStore: store,
CoverStore: store,
Producer: store,
Producer: producer,
TaskReader: store,
SearchIndex: searchIndex,
Kokoro: kokoroClient,
PocketTTS: pocketTTSClient,
Log: log,
},
)
@@ -165,3 +197,16 @@ func (n *noopKokoro) GenerateAudio(_ context.Context, _, _ string) ([]byte, erro
func (n *noopKokoro) ListVoices(_ context.Context) ([]string, error) {
return nil, nil
}
// parseRedisOpt converts a config.Redis into an asynq.RedisConnOpt.
// Handles full "redis://" / "rediss://" URLs and plain "host:port".
func parseRedisOpt(cfg config.Redis) (asynq.RedisConnOpt, error) {
addr := cfg.Addr
if len(addr) > 7 && (addr[:8] == "redis://" || (len(addr) > 8 && addr[:9] == "rediss://")) {
return asynq.ParseRedisURI(addr)
}
return asynq.RedisClientOpt{
Addr: addr,
Password: cfg.Password,
}, nil
}

View File

@@ -20,6 +20,7 @@ import (
"time"
"github.com/getsentry/sentry-go"
"github.com/libnovel/backend/internal/asynqqueue"
"github.com/libnovel/backend/internal/browser"
"github.com/libnovel/backend/internal/config"
"github.com/libnovel/backend/internal/kokoro"
@@ -29,6 +30,7 @@ import (
"github.com/libnovel/backend/internal/pockettts"
"github.com/libnovel/backend/internal/runner"
"github.com/libnovel/backend/internal/storage"
"github.com/libnovel/backend/internal/taskqueue"
)
// version and commit are set at build time via -ldflags.
@@ -150,10 +152,25 @@ func run() error {
OrchestratorWorkers: workers,
MetricsAddr: cfg.Runner.MetricsAddr,
CatalogueRefreshInterval: cfg.Runner.CatalogueRefreshInterval,
CatalogueRequestDelay: cfg.Runner.CatalogueRequestDelay,
SkipInitialCatalogueRefresh: cfg.Runner.SkipInitialCatalogueRefresh,
RedisAddr: cfg.Redis.Addr,
RedisPassword: cfg.Redis.Password,
}
// In Asynq mode the Consumer is a thin wrapper: claim/heartbeat/reap are
// no-ops, but FinishAudioTask / FinishScrapeTask / FailTask write back to
// PocketBase as before.
var consumer taskqueue.Consumer = store
if cfg.Redis.Addr != "" {
log.Info("runner: asynq mode — using Redis for task dispatch", "addr", cfg.Redis.Addr)
consumer = asynqqueue.NewConsumer(store)
} else {
log.Info("runner: poll mode — using PocketBase for task dispatch")
}
deps := runner.Dependencies{
Consumer: store,
Consumer: consumer,
BookWriter: store,
BookReader: store,
AudioStore: store,

View File

@@ -9,6 +9,7 @@ require (
require (
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
@@ -22,16 +23,25 @@ require (
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
github.com/hibiken/asynq v0.26.0 // indirect
github.com/hibiken/asynq/x v0.0.0-20260203063626-d704b68a426d // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/meilisearch/meilisearch-go v0.36.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/redis/go-redis/v9 v9.18.0 // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/tinylib/msgp v1.6.1 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/bridges/otelslog v0.17.0 // indirect
@@ -47,10 +57,12 @@ require (
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 // indirect
google.golang.org/grpc v1.79.2 // indirect

View File

@@ -1,5 +1,7 @@
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
@@ -27,6 +29,10 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=
github.com/hibiken/asynq v0.26.0 h1:1Zxr92MlDnb1Zt/QR5g2vSCqUS03i95lUfqx5X7/wrw=
github.com/hibiken/asynq v0.26.0/go.mod h1:Qk4e57bTnWDoyJ67VkchuV6VzSM9IQW2nPvAGuDyw58=
github.com/hibiken/asynq/x v0.0.0-20260203063626-d704b68a426d h1:Ld5m8EIK5QVOq/owOexKIbETij3skACg4eU1pArHsrw=
github.com/hibiken/asynq/x v0.0.0-20260203063626-d704b68a426d/go.mod h1:hhpStehaxSGg3ib9wJXzw5AXY1YS6lQ9BNavAgPbIhE=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
@@ -42,14 +48,36 @@ github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.98 h1:MeAVKjLVz+XJ28zFcuYyImNSAh8Mq725uNW4beRisi0=
github.com/minio/minio-go/v7 v7.0.98/go.mod h1:cY0Y+W7yozf0mdIclrttzo1Iiu7mEf9y7nk2uXqMOvM=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y=
github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.55.0 h1:KEi6DK7lXW/m7Ig5i47x0vRzuBsHuvJdi5ee6Y3G1dc=
github.com/prometheus/common v0.55.0/go.mod h1:2SECS4xJG1kd8XF9IcM1gMX6510RAEL65zxzNImwdc8=
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/redis/go-redis/v9 v9.18.0 h1:pMkxYPkEbMPwRdenAzUNyFNrDgHx9U+DrBabWNfSRQs=
github.com/redis/go-redis/v9 v9.18.0/go.mod h1:k3ufPphLU5YXwNTUcCRXGxUoF1fqxnhFQmscfkCoDA0=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
@@ -84,6 +112,8 @@ go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjce
go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
@@ -94,6 +124,8 @@ golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 h1:JLQynH/LBHfCTSbDWl+py8C+Rg/k1OVH3xfcaiANuF0=
google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:kSJwQxqmFXeo79zOmbrALdflXQeAYcUbgS7PbpMknCY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 h1:mWPCjDEyshlQYzBpMNHaEof6UX1PmHcaUODUywQ0uac=

View File

@@ -0,0 +1,56 @@
package asynqqueue
import (
"context"
"time"
"github.com/libnovel/backend/internal/domain"
"github.com/libnovel/backend/internal/taskqueue"
)
// Consumer wraps the PocketBase-backed Consumer for result write-back only.
//
// When using Asynq, the runner no longer polls for work — Asynq delivers
// tasks via the ServeMux handlers. The only Consumer operations the handlers
// need are:
// - FinishAudioTask / FinishScrapeTask — write result back to PocketBase
// - FailTask — mark PocketBase record as failed
//
// ClaimNextAudioTask, ClaimNextScrapeTask, HeartbeatTask, and ReapStaleTasks
// are all no-ops here because Asynq owns those responsibilities.
type Consumer struct {
pb taskqueue.Consumer // underlying PocketBase consumer (for write-back)
}
// NewConsumer wraps an existing PocketBase Consumer.
func NewConsumer(pb taskqueue.Consumer) *Consumer {
return &Consumer{pb: pb}
}
// ── Write-back (delegated to PocketBase) ──────────────────────────────────────
func (c *Consumer) FinishScrapeTask(ctx context.Context, id string, result domain.ScrapeResult) error {
return c.pb.FinishScrapeTask(ctx, id, result)
}
func (c *Consumer) FinishAudioTask(ctx context.Context, id string, result domain.AudioResult) error {
return c.pb.FinishAudioTask(ctx, id, result)
}
func (c *Consumer) FailTask(ctx context.Context, id, errMsg string) error {
return c.pb.FailTask(ctx, id, errMsg)
}
// ── No-ops (Asynq owns claiming / heartbeating / reaping) ───────────────────
func (c *Consumer) ClaimNextScrapeTask(_ context.Context, _ string) (domain.ScrapeTask, bool, error) {
return domain.ScrapeTask{}, false, nil
}
func (c *Consumer) ClaimNextAudioTask(_ context.Context, _ string) (domain.AudioTask, bool, error) {
return domain.AudioTask{}, false, nil
}
func (c *Consumer) HeartbeatTask(_ context.Context, _ string) error { return nil }
func (c *Consumer) ReapStaleTasks(_ context.Context, _ time.Duration) (int, error) { return 0, nil }

View File

@@ -0,0 +1,90 @@
package asynqqueue
import (
"context"
"encoding/json"
"fmt"
"github.com/hibiken/asynq"
"github.com/libnovel/backend/internal/taskqueue"
)
// Producer dual-writes every task: first to PocketBase (via pb, for audit /
// UI status), then to Redis via Asynq so the runner picks it up immediately.
type Producer struct {
pb taskqueue.Producer // underlying PocketBase producer
client *asynq.Client
}
// NewProducer wraps an existing PocketBase Producer with Asynq dispatch.
func NewProducer(pb taskqueue.Producer, redisOpt asynq.RedisConnOpt) *Producer {
return &Producer{
pb: pb,
client: asynq.NewClient(redisOpt),
}
}
// Close shuts down the underlying Asynq client connection.
func (p *Producer) Close() error {
return p.client.Close()
}
// CreateScrapeTask creates a PocketBase record then enqueues an Asynq job.
func (p *Producer) CreateScrapeTask(ctx context.Context, kind, targetURL string, fromChapter, toChapter int) (string, error) {
id, err := p.pb.CreateScrapeTask(ctx, kind, targetURL, fromChapter, toChapter)
if err != nil {
return "", err
}
payload := ScrapePayload{
PBTaskID: id,
Kind: kind,
TargetURL: targetURL,
FromChapter: fromChapter,
ToChapter: toChapter,
}
taskType := TypeScrapeBook
if kind == "catalogue" {
taskType = TypeScrapeCatalogue
}
if err := p.enqueue(ctx, taskType, payload); err != nil {
// Non-fatal: PB record exists; runner will pick it up on next poll.
return id, fmt.Errorf("asynq enqueue scrape (task still in PB): %w", err)
}
return id, nil
}
// CreateAudioTask creates a PocketBase record then enqueues an Asynq job.
func (p *Producer) CreateAudioTask(ctx context.Context, slug string, chapter int, voice string) (string, error) {
id, err := p.pb.CreateAudioTask(ctx, slug, chapter, voice)
if err != nil {
return "", err
}
payload := AudioPayload{
PBTaskID: id,
Slug: slug,
Chapter: chapter,
Voice: voice,
}
if err := p.enqueue(ctx, TypeAudioGenerate, payload); err != nil {
return id, fmt.Errorf("asynq enqueue audio (task still in PB): %w", err)
}
return id, nil
}
// CancelTask delegates to PocketBase; Asynq jobs may already be running and
// cannot be reliably cancelled, so we only update the audit record.
func (p *Producer) CancelTask(ctx context.Context, id string) error {
return p.pb.CancelTask(ctx, id)
}
// enqueue serialises payload and dispatches it to Asynq.
func (p *Producer) enqueue(_ context.Context, taskType string, payload any) error {
b, err := json.Marshal(payload)
if err != nil {
return fmt.Errorf("marshal payload: %w", err)
}
_, err = p.client.Enqueue(asynq.NewTask(taskType, b))
return err
}

View File

@@ -0,0 +1,46 @@
// Package asynqqueue provides Asynq-backed implementations of the
// taskqueue.Producer and taskqueue.Consumer interfaces.
//
// Architecture:
// - Producer: dual-writes — creates a PocketBase record for audit/UI, then
// enqueues an Asynq job so the runner picks it up immediately (sub-ms).
// - Consumer: thin wrapper used only for result write-back (FinishAudioTask,
// FinishScrapeTask, FailTask). ClaimNext*/Heartbeat/Reap are no-ops because
// Asynq owns those responsibilities.
// - Handlers: asynq.HandlerFunc wrappers that decode job payloads and invoke
// the existing runner logic (runScrapeTask / runAudioTask).
//
// Fallback: when REDIS_ADDR is empty the caller should use the plain
// storage.Store (PocketBase-polling) implementation unchanged.
package asynqqueue
// Queue names — keep all jobs on the default queue for now.
// Add separate queues (e.g. "audio", "scrape") later if you need priority.
const QueueDefault = "default"
// Task type constants used for Asynq routing.
const (
TypeAudioGenerate = "audio:generate"
TypeScrapeBook = "scrape:book"
TypeScrapeCatalogue = "scrape:catalogue"
)
// AudioPayload is the Asynq job payload for audio generation tasks.
type AudioPayload struct {
// PBTaskID is the PocketBase record ID created before enqueueing.
// The handler uses it to write results back via Consumer.FinishAudioTask.
PBTaskID string `json:"pb_task_id"`
Slug string `json:"slug"`
Chapter int `json:"chapter"`
Voice string `json:"voice"`
}
// ScrapePayload is the Asynq job payload for scrape tasks.
type ScrapePayload struct {
// PBTaskID is the PocketBase record ID created before enqueueing.
PBTaskID string `json:"pb_task_id"`
Kind string `json:"kind"` // "catalogue", "book", or "book_range"
TargetURL string `json:"target_url"` // empty for catalogue tasks
FromChapter int `json:"from_chapter"` // 0 unless Kind=="book_range"
ToChapter int `json:"to_chapter"` // 0 unless Kind=="book_range"
}

View File

@@ -47,6 +47,7 @@ import (
"github.com/libnovel/backend/internal/kokoro"
"github.com/libnovel/backend/internal/meili"
"github.com/libnovel/backend/internal/novelfire/htmlutil"
"github.com/libnovel/backend/internal/pockettts"
"github.com/libnovel/backend/internal/scraper"
)
@@ -703,7 +704,7 @@ func (s *Server) handleAudioProxy(w http.ResponseWriter, r *http.Request) {
// ── Voices ─────────────────────────────────────────────────────────────────────
// handleVoices handles GET /api/voices.
// Returns {"voices": [...]} — fetched from Kokoro with built-in fallback.
// Returns {"voices": [...]} — merged list from Kokoro and pocket-tts.
func (s *Server) handleVoices(w http.ResponseWriter, r *http.Request) {
writeJSON(w, 0, map[string]any{"voices": s.voices(r.Context())})
}
@@ -763,8 +764,8 @@ const voiceSampleText = "Hello! This is a preview of what I sound like. I hope y
// handlePresignVoiceSample handles GET /api/presign/voice-sample/{voice}.
// If the sample has not been generated yet it synthesises it on the fly via
// Kokoro, stores the result in MinIO, and returns the presigned URL — so the
// caller always gets a playable URL in a single request.
// the appropriate TTS engine (Kokoro for kokoro voices, pocket-tts for
// pocket-tts voices), stores the result in MinIO, and returns the presigned URL.
func (s *Server) handlePresignVoiceSample(w http.ResponseWriter, r *http.Request) {
voice := r.PathValue("voice")
if voice == "" {
@@ -777,7 +778,20 @@ func (s *Server) handlePresignVoiceSample(w http.ResponseWriter, r *http.Request
// Generate sample on demand when it is not in MinIO yet.
if !s.deps.AudioStore.AudioExists(r.Context(), key) {
s.deps.Log.Info("generating voice sample on demand", "voice", voice)
mp3, err := s.deps.Kokoro.GenerateAudio(r.Context(), voiceSampleText, voice)
var (
mp3 []byte
err error
)
if pockettts.IsPocketTTSVoice(voice) {
if s.deps.PocketTTS == nil {
jsonError(w, http.StatusServiceUnavailable, "pocket-tts not configured")
return
}
mp3, err = s.deps.PocketTTS.GenerateAudio(r.Context(), voiceSampleText, voice)
} else {
mp3, err = s.deps.Kokoro.GenerateAudio(r.Context(), voiceSampleText, voice)
}
if err != nil {
s.deps.Log.Error("voice sample generation failed", "voice", voice, "err", err)
jsonError(w, http.StatusInternalServerError, "voice sample generation failed")
@@ -1148,9 +1162,9 @@ func stripMarkdown(src string) string {
// ── Hardcoded Kokoro voice fallback ───────────────────────────────────────────
// kokoroVoices is the built-in fallback list used when the Kokoro service is
// unavailable. Matches the list in the old scraper helpers.go.
var kokoroVoices = []string{
// kokoroVoiceIDs is the built-in fallback list of Kokoro voice IDs used when
// the Kokoro service is unavailable.
var kokoroVoiceIDs = []string{
// American English
"af_alloy", "af_aoede", "af_bella", "af_heart", "af_jadzia",
"af_jessica", "af_kore", "af_nicole", "af_nova", "af_river",

View File

@@ -30,8 +30,10 @@ import (
sentryhttp "github.com/getsentry/sentry-go/http"
"github.com/libnovel/backend/internal/bookstore"
"github.com/libnovel/backend/internal/domain"
"github.com/libnovel/backend/internal/kokoro"
"github.com/libnovel/backend/internal/meili"
"github.com/libnovel/backend/internal/pockettts"
"github.com/libnovel/backend/internal/taskqueue"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
@@ -59,9 +61,12 @@ type Dependencies struct {
// SearchIndex provides full-text book search via Meilisearch.
// If nil, the local-only fallback search is used.
SearchIndex meili.Client
// Kokoro is the TTS client (used for voice list only in the backend;
// Kokoro is the Kokoro TTS client (used for voice list only in the backend;
// audio generation is done by the runner).
Kokoro kokoro.Client
// PocketTTS is the pocket-tts client (used for voice list only in the backend;
// audio generation is done by the runner).
PocketTTS pockettts.Client
// Log is the structured logger.
Log *slog.Logger
}
@@ -84,7 +89,7 @@ type Server struct {
// voiceMu guards cachedVoices. Populated lazily on first GET /api/voices.
voiceMu sync.RWMutex
cachedVoices []string
cachedVoices []domain.Voice
}
// New creates a Server from cfg and deps.
@@ -264,10 +269,10 @@ func jsonError(w http.ResponseWriter, status int, msg string) {
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
}
// voices returns the list of available Kokoro voices. On the first call it
// fetches from the Kokoro service and caches the result. Falls back to the
// hardcoded list on error.
func (s *Server) voices(ctx context.Context) []string {
// voices returns the merged list of available voices from Kokoro and pocket-tts.
// On the first call it fetches from both services and caches the result.
// Falls back to the hardcoded Kokoro list on error.
func (s *Server) voices(ctx context.Context) []domain.Voice {
s.voiceMu.RLock()
cached := s.cachedVoices
s.voiceMu.RUnlock()
@@ -275,23 +280,89 @@ func (s *Server) voices(ctx context.Context) []string {
return cached
}
if s.deps.Kokoro == nil {
return kokoroVoices
}
fetchCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
list, err := s.deps.Kokoro.ListVoices(fetchCtx)
if err != nil || len(list) == 0 {
s.deps.Log.Warn("backend: could not fetch kokoro voices, using built-in list", "err", err)
return kokoroVoices
var result []domain.Voice
// ── Kokoro voices ─────────────────────────────────────────────────────────
var kokoroIDs []string
if s.deps.Kokoro != nil {
ids, err := s.deps.Kokoro.ListVoices(fetchCtx)
if err != nil || len(ids) == 0 {
s.deps.Log.Warn("backend: could not fetch kokoro voices, using built-in list", "err", err)
ids = kokoroVoiceIDs
} else {
s.deps.Log.Info("backend: fetched kokoro voices", "count", len(ids))
}
kokoroIDs = ids
} else {
kokoroIDs = kokoroVoiceIDs
}
for _, id := range kokoroIDs {
result = append(result, kokoroVoice(id))
}
// ── Pocket-TTS voices ─────────────────────────────────────────────────────
if s.deps.PocketTTS != nil {
ids, err := s.deps.PocketTTS.ListVoices(fetchCtx)
if err != nil {
s.deps.Log.Warn("backend: could not fetch pocket-tts voices", "err", err)
} else {
for _, id := range ids {
result = append(result, pocketTTSVoice(id))
}
s.deps.Log.Info("backend: fetched pocket-tts voices", "count", len(ids))
}
}
s.voiceMu.Lock()
s.cachedVoices = list
s.cachedVoices = result
s.voiceMu.Unlock()
s.deps.Log.Info("backend: fetched kokoro voices", "count", len(list))
return list
return result
}
// kokoroVoice builds a domain.Voice for a Kokoro voice ID.
// The two-character prefix encodes language and gender:
//
// af/am → en-us f/m | bf/bm → en-gb f/m
// ef/em → es f/m | ff → fr f
// hf/hm → hi f/m | if/im → it f/m
// jf/jm → ja f/m | pf/pm → pt f/m
// zf/zm → zh f/m
func kokoroVoice(id string) domain.Voice {
type meta struct{ lang, gender string }
prefixMap := map[string]meta{
"af": {"en-us", "f"}, "am": {"en-us", "m"},
"bf": {"en-gb", "f"}, "bm": {"en-gb", "m"},
"ef": {"es", "f"}, "em": {"es", "m"},
"ff": {"fr", "f"},
"hf": {"hi", "f"}, "hm": {"hi", "m"},
"if": {"it", "f"}, "im": {"it", "m"},
"jf": {"ja", "f"}, "jm": {"ja", "m"},
"pf": {"pt", "f"}, "pm": {"pt", "m"},
"zf": {"zh", "f"}, "zm": {"zh", "m"},
}
if len(id) >= 2 {
if m, ok := prefixMap[id[:2]]; ok {
return domain.Voice{ID: id, Engine: "kokoro", Lang: m.lang, Gender: m.gender}
}
}
return domain.Voice{ID: id, Engine: "kokoro", Lang: "en", Gender: ""}
}
// pocketTTSVoice builds a domain.Voice for a pocket-tts voice ID.
// All pocket-tts voices are English audiobook narrators.
func pocketTTSVoice(id string) domain.Voice {
femaleVoices := map[string]struct{}{
"alba": {}, "fantine": {}, "cosette": {}, "eponine": {},
"azelma": {}, "anna": {}, "vera": {}, "mary": {}, "jane": {}, "eve": {},
}
gender := "m"
if _, ok := femaleVoices[id]; ok {
gender = "f"
}
return domain.Voice{ID: id, Engine: "pocket-tts", Lang: "en", Gender: gender}
}
// handleHealth handles GET /health.

View File

@@ -86,6 +86,19 @@ type Valkey struct {
Addr string
}
// Redis holds connection settings for the Asynq task queue Redis instance.
// This is separate from Valkey (presign cache) — it may point to the same
// Redis or a dedicated one. An empty Addr falls back to PocketBase polling.
type Redis struct {
// Addr is the host:port (or rediss://... URL) of the Redis instance.
// Use rediss:// scheme for TLS (e.g. rediss://:password@redis.libnovel.cc:6380).
// An empty string disables Asynq and falls back to PocketBase polling.
Addr string
// Password is the Redis AUTH password.
// Not needed when Addr is a full rediss:// URL that includes the password.
Password string
}
// Runner holds settings specific to the runner/worker binary.
type Runner struct {
// PollInterval is how often the runner checks PocketBase for pending tasks.
@@ -113,6 +126,11 @@ type Runner struct {
// is already indexed and a 24h walk would be wasteful.
// Controlled by RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH=true.
SkipInitialCatalogueRefresh bool
// CatalogueRequestDelay is the base delay inserted between per-book metadata
// requests during a catalogue refresh. A random jitter of up to 50% is added
// on top. Defaults to 2s. Increase to reduce 429 pressure on novelfire.net.
// Controlled by RUNNER_CATALOGUE_REQUEST_DELAY (e.g. "3s", "500ms").
CatalogueRequestDelay time.Duration
}
// Config is the top-level configuration struct consumed by both binaries.
@@ -125,6 +143,7 @@ type Config struct {
Runner Runner
Meilisearch Meilisearch
Valkey Valkey
Redis Redis
// LogLevel is one of "debug", "info", "warn", "error".
LogLevel string
}
@@ -182,6 +201,7 @@ func Load() Config {
MetricsAddr: envOr("RUNNER_METRICS_ADDR", ":9091"),
CatalogueRefreshInterval: envDuration("RUNNER_CATALOGUE_REFRESH_INTERVAL", 0),
SkipInitialCatalogueRefresh: envBool("RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH", false),
CatalogueRequestDelay: envDuration("RUNNER_CATALOGUE_REQUEST_DELAY", 2*time.Second),
},
Meilisearch: Meilisearch{
@@ -192,6 +212,11 @@ func Load() Config {
Valkey: Valkey{
Addr: envOr("VALKEY_ADDR", ""),
},
Redis: Redis{
Addr: envOr("REDIS_ADDR", ""),
Password: envOr("REDIS_PASSWORD", ""),
},
}
}

View File

@@ -60,6 +60,20 @@ type RankingItem struct {
Updated time.Time `json:"updated,omitempty"`
}
// ── Voice types ───────────────────────────────────────────────────────────────
// Voice describes a single text-to-speech voice available in the system.
type Voice struct {
// ID is the voice identifier passed to TTS clients (e.g. "af_bella", "alba").
ID string `json:"id"`
// Engine is "kokoro" or "pocket-tts".
Engine string `json:"engine"`
// Lang is the primary language tag (e.g. "en-us", "en-gb", "en", "es", "fr").
Lang string `json:"lang"`
// Gender is "f" or "m".
Gender string `json:"gender"`
}
// ── Storage record types ──────────────────────────────────────────────────────
// ChapterInfo is a lightweight chapter descriptor stored in the index.

View File

@@ -13,6 +13,7 @@ import (
"errors"
"fmt"
"log/slog"
"math/rand"
"net/url"
"path"
"strconv"
@@ -55,6 +56,9 @@ func (s *Scraper) SourceName() string { return "novelfire.net" }
// ── CatalogueProvider ─────────────────────────────────────────────────────────
// ScrapeCatalogue streams all CatalogueEntry values across all catalogue pages.
// Each page fetch uses retryGet with 429-aware exponential backoff.
// A small inter-page delay (cataloguePageDelay) is inserted between requests to
// avoid hammering the server when paging through hundreds of catalogue pages.
func (s *Scraper) ScrapeCatalogue(ctx context.Context) (<-chan domain.CatalogueEntry, <-chan error) {
entries := make(chan domain.CatalogueEntry, 64)
errs := make(chan error, 16)
@@ -73,8 +77,18 @@ func (s *Scraper) ScrapeCatalogue(ctx context.Context) (<-chan domain.CatalogueE
default:
}
// Polite inter-page delay — skipped on the very first page.
if page > 1 {
jitter := time.Duration(500+rand.Intn(1000)) * time.Millisecond
select {
case <-ctx.Done():
return
case <-time.After(jitter):
}
}
s.log.Info("scraping catalogue page", "page", page, "url", pageURL)
raw, err := s.client.GetContent(ctx, pageURL)
raw, err := retryGet(ctx, s.log, s.client, pageURL, 9, 10*time.Second)
if err != nil {
errs <- fmt.Errorf("catalogue page %d: %w", page, err)
return
@@ -139,10 +153,11 @@ func (s *Scraper) ScrapeCatalogue(ctx context.Context) (<-chan domain.CatalogueE
// ── MetadataProvider ──────────────────────────────────────────────────────────
// ScrapeMetadata fetches and parses book metadata from the book's landing page.
// Uses retryGet with 429-aware exponential backoff (up to 9 attempts).
func (s *Scraper) ScrapeMetadata(ctx context.Context, bookURL string) (domain.BookMeta, error) {
s.log.Debug("metadata fetch starting", "url", bookURL)
raw, err := s.client.GetContent(ctx, bookURL)
raw, err := retryGet(ctx, s.log, s.client, bookURL, 9, 10*time.Second)
if err != nil {
return domain.BookMeta{}, fmt.Errorf("metadata fetch %s: %w", bookURL, err)
}

View File

@@ -0,0 +1,149 @@
package runner
// asynq_runner.go — Asynq-based task dispatch for the runner.
//
// When cfg.RedisAddr is set, Run() calls runAsynq() instead of runPoll().
// The Asynq server replaces the polling loop: it listens on Redis for tasks
// enqueued by the backend Producer and delivers them immediately.
//
// Handlers in this file decode Asynq job payloads and call the existing
// runScrapeTask / runAudioTask methods, keeping all execution logic in one place.
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/hibiken/asynq"
asynqmetrics "github.com/hibiken/asynq/x/metrics"
"github.com/libnovel/backend/internal/asynqqueue"
"github.com/libnovel/backend/internal/domain"
)
// runAsynq starts an Asynq server that replaces the PocketBase poll loop.
// It also starts the periodic catalogue refresh ticker.
// Blocks until ctx is cancelled.
func (r *Runner) runAsynq(ctx context.Context) error {
redisOpt, err := r.redisConnOpt()
if err != nil {
return fmt.Errorf("runner: parse redis addr: %w", err)
}
srv := asynq.NewServer(redisOpt, asynq.Config{
// Allocate concurrency slots for each task type.
// Total concurrency = scrape + audio slots.
Concurrency: r.cfg.MaxConcurrentScrape + r.cfg.MaxConcurrentAudio,
Queues: map[string]int{
asynqqueue.QueueDefault: 1,
},
// Let Asynq handle retries with exponential back-off.
RetryDelayFunc: asynq.DefaultRetryDelayFunc,
// Log errors from handlers via the existing structured logger.
ErrorHandler: asynq.ErrorHandlerFunc(func(_ context.Context, task *asynq.Task, err error) {
r.deps.Log.Error("runner: asynq task failed",
"type", task.Type(),
"err", err,
)
}),
})
mux := asynq.NewServeMux()
mux.HandleFunc(asynqqueue.TypeAudioGenerate, r.handleAudioTask)
mux.HandleFunc(asynqqueue.TypeScrapeBook, r.handleScrapeTask)
mux.HandleFunc(asynqqueue.TypeScrapeCatalogue, r.handleScrapeTask)
// Register Asynq queue metrics with the default Prometheus registry so
// the /metrics endpoint (metrics.go) can expose them.
inspector := asynq.NewInspector(redisOpt)
collector := asynqmetrics.NewQueueMetricsCollector(inspector)
if err := r.metricsRegistry.Register(collector); err != nil {
r.deps.Log.Warn("runner: could not register asynq prometheus collector", "err", err)
}
// Start the periodic catalogue refresh.
catalogueTick := time.NewTicker(r.cfg.CatalogueRefreshInterval)
defer catalogueTick.Stop()
if !r.cfg.SkipInitialCatalogueRefresh {
go r.runCatalogueRefresh(ctx)
} else {
r.deps.Log.Info("runner: skipping initial catalogue refresh (RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH=true)")
}
r.deps.Log.Info("runner: asynq mode active", "redis_addr", r.cfg.RedisAddr)
// Run catalogue refresh ticker in the background.
go func() {
for {
select {
case <-ctx.Done():
return
case <-catalogueTick.C:
go r.runCatalogueRefresh(ctx)
}
}
}()
// Start Asynq server (non-blocking).
if err := srv.Start(mux); err != nil {
return fmt.Errorf("runner: asynq server start: %w", err)
}
// Block until context is cancelled, then gracefully stop.
<-ctx.Done()
r.deps.Log.Info("runner: context cancelled, shutting down asynq server")
srv.Shutdown()
return nil
}
// redisConnOpt parses cfg.RedisAddr into an asynq.RedisConnOpt.
// Supports full "redis://" / "rediss://" URLs and plain "host:port".
func (r *Runner) redisConnOpt() (asynq.RedisConnOpt, error) {
addr := r.cfg.RedisAddr
// ParseRedisURI handles redis:// and rediss:// schemes.
if len(addr) > 7 && (addr[:8] == "redis://" || addr[:9] == "rediss://") {
return asynq.ParseRedisURI(addr)
}
// Plain "host:port" — use RedisClientOpt directly.
return asynq.RedisClientOpt{
Addr: addr,
Password: r.cfg.RedisPassword,
}, nil
}
// handleScrapeTask is the Asynq handler for TypeScrapeBook and TypeScrapeCatalogue.
func (r *Runner) handleScrapeTask(ctx context.Context, t *asynq.Task) error {
var p asynqqueue.ScrapePayload
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return fmt.Errorf("unmarshal scrape payload: %w", err)
}
task := domain.ScrapeTask{
ID: p.PBTaskID,
Kind: p.Kind,
TargetURL: p.TargetURL,
FromChapter: p.FromChapter,
ToChapter: p.ToChapter,
}
r.tasksRunning.Add(1)
defer r.tasksRunning.Add(-1)
r.runScrapeTask(ctx, task)
return nil
}
// handleAudioTask is the Asynq handler for TypeAudioGenerate.
func (r *Runner) handleAudioTask(ctx context.Context, t *asynq.Task) error {
var p asynqqueue.AudioPayload
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return fmt.Errorf("unmarshal audio payload: %w", err)
}
task := domain.AudioTask{
ID: p.PBTaskID,
Slug: p.Slug,
Chapter: p.Chapter,
Voice: p.Voice,
}
r.tasksRunning.Add(1)
defer r.tasksRunning.Add(-1)
r.runAudioTask(ctx, task)
return nil
}

View File

@@ -6,17 +6,20 @@ package runner
//
// Design:
// - Runs on its own ticker (CatalogueRefreshInterval, default 24h) inside Run().
// - Also fires once on startup.
// - ScrapeCatalogue streams CatalogueEntry values over a channel — we iterate
// and call ScrapeMetadata for each entry.
// - Per-request random jitter (13s) prevents hammering novelfire.net.
// - Cover images are fetched from the URL embedded in BookMeta.Cover and
// stored in MinIO (browse bucket, key: covers/{slug}.jpg).
// - WriteMetadata + UpsertBook are called for every successfully scraped book.
// - Errors for individual books are logged and skipped; the loop continues.
// - The cover URL stored in BookMeta.Cover is rewritten to the internal proxy
// path (/api/cover/novelfire.net/{slug}) so the UI always fetches via the
// backend, which will serve from MinIO.
// - Also fires once on startup (unless SkipInitialCatalogueRefresh is set).
// - ScrapeCatalogue streams CatalogueEntry values over a channel — already has
// its own inter-page jitter + retryGet (see scraper.go).
// - Per-book: only metadata is scraped here (not chapters). Chapters are scraped
// on-demand when a user opens a book or via an explicit scrape task.
// - Between each metadata request a configurable base delay plus up to 50%
// random jitter is applied (CatalogueRequestDelay, default 2s). This keeps
// the request rate well below novelfire.net's rate limit even for ~15k books.
// - ScrapeMetadata itself uses retryGet with 429-aware exponential backoff
// (up to 9 attempts), so transient rate limits are handled gracefully.
// - Cover images are fetched and stored in MinIO on first sight; subsequent
// refreshes skip covers that already exist (CoverExists check).
// - Books already present in Meilisearch are skipped entirely (fast path).
// - Errors for individual books are logged and skipped; the loop never aborts.
import (
"context"
@@ -29,7 +32,7 @@ import (
// runCatalogueRefresh performs one full catalogue walk: scrapes metadata for
// every book on novelfire.net, downloads covers to MinIO, and upserts to
// Meilisearch. Errors for individual books are logged and skipped.
// Meilisearch. Individual book failures are logged and skipped.
func (r *Runner) runCatalogueRefresh(ctx context.Context) {
if r.deps.Novel == nil {
r.deps.Log.Warn("runner: catalogue refresh skipped — Novel scraper not configured")
@@ -40,8 +43,9 @@ func (r *Runner) runCatalogueRefresh(ctx context.Context) {
return
}
delay := r.cfg.CatalogueRequestDelay
log := r.deps.Log.With("op", "catalogue_refresh")
log.Info("runner: catalogue refresh starting")
log.Info("runner: catalogue refresh starting", "request_delay", delay)
entries, errCh := r.deps.Novel.ScrapeCatalogue(ctx)
@@ -51,26 +55,26 @@ func (r *Runner) runCatalogueRefresh(ctx context.Context) {
break
}
// Skip books already present in Meilisearch — they were indexed on a
// previous run. Re-indexing only happens when a scrape task is
// explicitly enqueued (e.g. via the admin UI or API).
// Fast path: skip books already indexed in Meilisearch.
if r.deps.SearchIndex.BookExists(ctx, entry.Slug) {
skipped++
continue
}
// Random jitter between books to avoid rate-limiting.
jitter := time.Duration(1000+rand.Intn(2000)) * time.Millisecond
// Polite delay between metadata requests: base + up to 50% jitter.
// This applies before every fetch so we never fire bursts.
jitter := time.Duration(rand.Int63n(int64(delay / 2)))
select {
case <-ctx.Done():
break
case <-time.After(jitter):
case <-time.After(delay + jitter):
}
// ScrapeMetadata internally retries on 429 with exponential back-off.
meta, err := r.deps.Novel.ScrapeMetadata(ctx, entry.URL)
if err != nil {
log.Warn("runner: catalogue refresh: metadata scrape failed",
"url", entry.URL, "err", err)
log.Warn("runner: catalogue refresh: metadata scrape failed — skipping book",
"slug", entry.Slug, "url", entry.URL, "err", err)
errCount++
continue
}
@@ -81,35 +85,32 @@ func (r *Runner) runCatalogueRefresh(ctx context.Context) {
// Persist to PocketBase.
if err := r.deps.BookWriter.WriteMetadata(ctx, meta); err != nil {
log.Warn("runner: catalogue refresh: WriteMetadata failed",
log.Warn("runner: catalogue refresh: WriteMetadata failed — skipping book",
"slug", meta.Slug, "err", err)
errCount++
continue
}
// Index in Meilisearch.
// Index in Meilisearch (non-fatal).
if err := r.deps.SearchIndex.UpsertBook(ctx, meta); err != nil {
log.Warn("runner: catalogue refresh: UpsertBook failed",
"slug", meta.Slug, "err", err)
// non-fatal — continue
}
// Download and store cover image in MinIO if we have a cover URL
// and a CoverStore is wired in.
// Download cover to MinIO if not already cached (non-fatal).
if r.deps.CoverStore != nil && originalCover != "" {
if !r.deps.CoverStore.CoverExists(ctx, meta.Slug) {
if err := r.downloadCover(ctx, meta.Slug, originalCover); err != nil {
log.Warn("runner: catalogue refresh: cover download failed",
"slug", meta.Slug, "url", originalCover, "err", err)
// non-fatal
}
}
}
ok++
if ok%100 == 0 {
if ok%50 == 0 {
log.Info("runner: catalogue refresh progress",
"scraped", ok, "errors", errCount)
"scraped", ok, "skipped", skipped, "errors", errCount)
}
}

View File

@@ -1,21 +1,28 @@
package runner
// metrics.go — lightweight HTTP metrics endpoint for the runner.
// metrics.go — Prometheus metrics HTTP endpoint for the runner.
//
// GET /metrics returns a JSON document with live task counters and uptime.
// No external dependency (no Prometheus); plain net/http only.
// GET /metrics returns a Prometheus text/plain scrape response.
// Exposes:
// - Standard Go runtime metrics (via promhttp)
// - Runner task counters (tasks_running, tasks_completed, tasks_failed)
// - Asynq queue metrics (registered in asynq_runner.go when Redis is enabled)
//
// GET /health — simple liveness probe.
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net"
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
// metricsServer serves GET /metrics for the runner process.
// metricsServer serves GET /metrics and GET /health for the runner process.
type metricsServer struct {
addr string
r *Runner
@@ -23,21 +30,62 @@ type metricsServer struct {
}
func newMetricsServer(addr string, r *Runner, log *slog.Logger) *metricsServer {
return &metricsServer{addr: addr, r: r, log: log}
ms := &metricsServer{addr: addr, r: r, log: log}
ms.registerCollectors()
return ms
}
// registerCollectors registers runner-specific Prometheus collectors.
// Called once at construction; Asynq queue collector is registered separately
// in asynq_runner.go after the Redis connection is established.
func (ms *metricsServer) registerCollectors() {
// Runner task gauges / counters backed by the atomic fields on Runner.
ms.r.metricsRegistry.MustRegister(prometheus.NewGaugeFunc(
prometheus.GaugeOpts{
Namespace: "runner",
Name: "tasks_running",
Help: "Number of tasks currently being processed.",
},
func() float64 { return float64(ms.r.tasksRunning.Load()) },
))
ms.r.metricsRegistry.MustRegister(prometheus.NewCounterFunc(
prometheus.CounterOpts{
Namespace: "runner",
Name: "tasks_completed_total",
Help: "Total number of tasks completed successfully since startup.",
},
func() float64 { return float64(ms.r.tasksCompleted.Load()) },
))
ms.r.metricsRegistry.MustRegister(prometheus.NewCounterFunc(
prometheus.CounterOpts{
Namespace: "runner",
Name: "tasks_failed_total",
Help: "Total number of tasks that ended in failure since startup.",
},
func() float64 { return float64(ms.r.tasksFailed.Load()) },
))
ms.r.metricsRegistry.MustRegister(prometheus.NewGaugeFunc(
prometheus.GaugeOpts{
Namespace: "runner",
Name: "uptime_seconds",
Help: "Seconds since the runner process started.",
},
func() float64 { return time.Since(ms.r.startedAt).Seconds() },
))
}
// ListenAndServe starts the HTTP server and blocks until ctx is cancelled or
// a fatal listen error occurs.
func (ms *metricsServer) ListenAndServe(ctx context.Context) error {
mux := http.NewServeMux()
mux.HandleFunc("GET /metrics", ms.handleMetrics)
mux.Handle("GET /metrics", promhttp.HandlerFor(ms.r.metricsRegistry, promhttp.HandlerOpts{}))
mux.HandleFunc("GET /health", ms.handleHealth)
srv := &http.Server{
Addr: ms.addr,
Handler: mux,
ReadTimeout: 5 * time.Second,
WriteTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
BaseContext: func(_ net.Listener) context.Context { return ctx },
}
@@ -58,35 +106,8 @@ func (ms *metricsServer) ListenAndServe(ctx context.Context) error {
}
}
// handleMetrics handles GET /metrics.
// Response shape (JSON):
//
// {
// "tasks_running": N,
// "tasks_completed": N,
// "tasks_failed": N,
// "uptime_seconds": N
// }
func (ms *metricsServer) handleMetrics(w http.ResponseWriter, _ *http.Request) {
uptimeSec := int64(time.Since(ms.r.startedAt).Seconds())
metricsWriteJSON(w, 0, map[string]int64{
"tasks_running": ms.r.tasksRunning.Load(),
"tasks_completed": ms.r.tasksCompleted.Load(),
"tasks_failed": ms.r.tasksFailed.Load(),
"uptime_seconds": uptimeSec,
})
}
// handleHealth handles GET /health — simple liveness probe for the metrics server.
// handleHealth handles GET /health — simple liveness probe.
func (ms *metricsServer) handleHealth(w http.ResponseWriter, _ *http.Request) {
metricsWriteJSON(w, 0, map[string]string{"status": "ok"})
}
// metricsWriteJSON writes v as a JSON response with the given status code.
func metricsWriteJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
if status != 0 {
w.WriteHeader(status)
}
_ = json.NewEncoder(w).Encode(v)
_, _ = w.Write([]byte(`{"status":"ok"}`))
}

View File

@@ -34,6 +34,7 @@ import (
"github.com/libnovel/backend/internal/pockettts"
"github.com/libnovel/backend/internal/scraper"
"github.com/libnovel/backend/internal/taskqueue"
"github.com/prometheus/client_golang/prometheus"
)
// Config tunes the runner behaviour.
@@ -41,6 +42,7 @@ type Config struct {
// WorkerID uniquely identifies this runner instance in PocketBase records.
WorkerID string
// PollInterval is how often the runner checks for new tasks.
// Only used in PocketBase-polling mode (RedisAddr == "").
PollInterval time.Duration
// MaxConcurrentScrape limits simultaneous book-scrape goroutines.
MaxConcurrentScrape int
@@ -50,14 +52,20 @@ type Config struct {
OrchestratorWorkers int
// HeartbeatInterval is how often active tasks PATCH their heartbeat_at
// timestamp to signal they are still alive. Defaults to 30s when 0.
// Only used in PocketBase-polling mode.
HeartbeatInterval time.Duration
// StaleTaskThreshold is how old a heartbeat must be (or absent) before the
// task is considered orphaned and reset to pending. Defaults to 2m when 0.
// Only used in PocketBase-polling mode.
StaleTaskThreshold time.Duration
// CatalogueRefreshInterval is how often the runner walks the full catalogue,
// scrapes per-book metadata, downloads covers, and re-indexes everything in
// Meilisearch. Defaults to 24h (expensive — full catalogue walk).
CatalogueRefreshInterval time.Duration
// CatalogueRequestDelay is the base inter-request pause during a catalogue
// refresh metadata walk. Jitter of up to 50% is added on top.
// Defaults to 2s. Set via RUNNER_CATALOGUE_REQUEST_DELAY.
CatalogueRequestDelay time.Duration
// SkipInitialCatalogueRefresh suppresses the immediate catalogue walk that
// otherwise fires at startup. The periodic ticker (CatalogueRefreshInterval)
// still fires normally. Set RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH=true for
@@ -66,6 +74,15 @@ type Config struct {
// MetricsAddr is the HTTP listen address for the /metrics endpoint.
// Defaults to ":9091". Set to "" to disable.
MetricsAddr string
// RedisAddr is the address of the Redis instance used for Asynq task
// dispatch. When set the runner switches from PocketBase-polling mode to
// Asynq ServeMux mode (immediate task delivery, no polling).
// Supports plain "host:port" or a full "rediss://..." URL.
// When empty the runner falls back to PocketBase polling.
RedisAddr string
// RedisPassword is the Redis AUTH password.
// Not required when RedisAddr is a full URL that includes credentials.
RedisPassword string
}
// Dependencies are the external services the runner depends on.
@@ -99,6 +116,8 @@ type Runner struct {
cfg Config
deps Dependencies
metricsRegistry *prometheus.Registry
// Atomic task counters — read by /metrics without locking.
tasksRunning atomic.Int64
tasksCompleted atomic.Int64
@@ -130,6 +149,9 @@ func New(cfg Config, deps Dependencies) *Runner {
if cfg.CatalogueRefreshInterval <= 0 {
cfg.CatalogueRefreshInterval = 24 * time.Hour
}
if cfg.CatalogueRequestDelay <= 0 {
cfg.CatalogueRequestDelay = 2 * time.Second
}
if cfg.MetricsAddr == "" {
cfg.MetricsAddr = ":9091"
}
@@ -139,15 +161,18 @@ func New(cfg Config, deps Dependencies) *Runner {
if deps.SearchIndex == nil {
deps.SearchIndex = meili.NoopClient{}
}
return &Runner{cfg: cfg, deps: deps, startedAt: time.Now()}
return &Runner{cfg: cfg, deps: deps, startedAt: time.Now(), metricsRegistry: prometheus.NewRegistry()}
}
// Run starts the poll loop and the metrics HTTP server, blocking until ctx is
// cancelled.
// Run starts the worker loop and the metrics HTTP server, blocking until ctx
// is cancelled.
//
// When cfg.RedisAddr is set the runner uses Asynq (immediate task delivery).
// Otherwise it falls back to PocketBase polling (legacy mode).
func (r *Runner) Run(ctx context.Context) error {
r.deps.Log.Info("runner: starting",
"worker_id", r.cfg.WorkerID,
"poll_interval", r.cfg.PollInterval,
"mode", r.mode(),
"max_scrape", r.cfg.MaxConcurrentScrape,
"max_audio", r.cfg.MaxConcurrentAudio,
"catalogue_refresh_interval", r.cfg.CatalogueRefreshInterval,
@@ -164,6 +189,23 @@ func (r *Runner) Run(ctx context.Context) error {
}()
}
if r.cfg.RedisAddr != "" {
return r.runAsynq(ctx)
}
return r.runPoll(ctx)
}
// mode returns a short string describing the active dispatch mode.
func (r *Runner) mode() string {
if r.cfg.RedisAddr != "" {
return "asynq"
}
return "poll"
}
// runPoll is the legacy PocketBase-polling dispatch loop.
// Used when cfg.RedisAddr is empty.
func (r *Runner) runPoll(ctx context.Context) error {
scrapeSem := make(chan struct{}, r.cfg.MaxConcurrentScrape)
audioSem := make(chan struct{}, r.cfg.MaxConcurrentAudio)
var wg sync.WaitGroup
@@ -181,6 +223,8 @@ func (r *Runner) Run(ctx context.Context) error {
r.deps.Log.Info("runner: skipping initial catalogue refresh (RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH=true)")
}
r.deps.Log.Info("runner: poll mode active", "poll_interval", r.cfg.PollInterval)
// Run one poll immediately on startup, then on each tick.
for {
r.poll(ctx, scrapeSem, audioSem, &wg)

View File

@@ -706,7 +706,7 @@ func (s *Store) ListAudioTasks(ctx context.Context) ([]domain.AudioTask, error)
}
func (s *Store) GetAudioTask(ctx context.Context, cacheKey string) (domain.AudioTask, bool, error) {
filter := fmt.Sprintf(`cache_key=%q`, cacheKey)
filter := fmt.Sprintf(`cache_key='%s'`, cacheKey)
items, err := s.pb.listAll(ctx, "audio_jobs", filter, "-started")
if err != nil || len(items) == 0 {
return domain.AudioTask{}, false, err

View File

@@ -2,7 +2,8 @@ FROM caddy:2-builder AS builder
RUN xcaddy build \
--with github.com/mholt/caddy-ratelimit \
--with github.com/hslatman/caddy-crowdsec-bouncer/http
--with github.com/hslatman/caddy-crowdsec-bouncer/http \
--with github.com/mholt/caddy-l4
FROM caddy:2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

View File

@@ -15,7 +15,7 @@ x-infra-env: &infra-env
POCKETBASE_ADMIN_EMAIL: "${POCKETBASE_ADMIN_EMAIL}"
POCKETBASE_ADMIN_PASSWORD: "${POCKETBASE_ADMIN_PASSWORD}"
# Meilisearch
MEILI_URL: "http://meilisearch:7700"
MEILI_URL: "${MEILI_URL:-http://meilisearch:7700}"
MEILI_API_KEY: "${MEILI_MASTER_KEY}"
# Valkey
VALKEY_ADDR: "valkey:6379"
@@ -154,15 +154,21 @@ services:
# No public port — all traffic is routed via Caddy.
expose:
- "8080"
environment:
environment:
<<: *infra-env
BACKEND_HTTP_ADDR: ":8080"
LOG_LEVEL: "${LOG_LEVEL}"
KOKORO_URL: "${KOKORO_URL}"
KOKORO_VOICE: "${KOKORO_VOICE}"
POCKET_TTS_URL: "${POCKET_TTS_URL}"
GLITCHTIP_DSN: "${GLITCHTIP_DSN}"
OTEL_EXPORTER_OTLP_ENDPOINT: "${OTEL_EXPORTER_OTLP_ENDPOINT}"
OTEL_SERVICE_NAME: "backend"
# Asynq task queue — backend enqueues jobs to homelab Redis via Caddy TLS proxy.
# Set to "rediss://:password@redis.libnovel.cc:6380" in Doppler prd config.
# Leave empty to fall back to PocketBase polling.
REDIS_ADDR: "${REDIS_ADDR}"
REDIS_PASSWORD: "${REDIS_PASSWORD}"
healthcheck:
test: ["CMD", "/healthcheck", "http://localhost:8080/health"]
interval: 15s
@@ -353,13 +359,16 @@ services:
# ─── Caddy (reverse proxy + automatic HTTPS) ──────────────────────────────────
# Custom build includes github.com/mholt/caddy-ratelimit and
# github.com/hslatman/caddy-crowdsec-bouncer/http.
# Custom build includes github.com/mholt/caddy-ratelimit,
# github.com/hslatman/caddy-crowdsec-bouncer/http, and
# github.com/mholt/caddy-l4 (TCP layer4 proxy for Redis).
caddy:
image: kalekber/libnovel-caddy:${GIT_TAG:-latest}
build:
context: ./caddy
dockerfile: Dockerfile
labels:
com.centurylinklabs.watchtower.enable: "true"
restart: unless-stopped
depends_on:
backend:
@@ -372,9 +381,12 @@ services:
- "80:80"
- "443:443"
- "443:443/udp" # HTTP/3 (QUIC)
- "6380:6380" # Redis TCP proxy (TLS) for homelab → Asynq
environment:
DOMAIN: "${DOMAIN}"
CADDY_ACME_EMAIL: "${CADDY_ACME_EMAIL}"
# Homelab Redis address — Caddy TCP-proxies inbound :6380 to this.
HOMELAB_REDIS_ADDR: "${HOMELAB_REDIS_ADDR:?HOMELAB_REDIS_ADDR required for Redis TCP proxy}"
env_file:
- path: ./crowdsec/.crowdsec.env
required: false

View File

@@ -394,7 +394,7 @@ services:
# Voices match existing IDs: af_bella, af_sky, af_heart, etc.
# The runner reaches it at http://kokoro-fastapi:8880 via the Docker network.
kokoro-fastapi:
image: ghcr.io/remsky/kokoro-fastapi-gpu:latest
image: kokoro-fastapi:latest
restart: unless-stopped
deploy:
resources:

View File

@@ -0,0 +1,16 @@
# Grafana alerting provisioning — contact points
# Sends all alerts to Gotify (self-hosted push notifications).
apiVersion: 1
contactPoints:
- orgId: 1
name: Gotify
receivers:
- uid: gotify-webhook
type: webhook
settings:
url: "http://gotify/message?token=ABZrZgCY-4ivcmt"
httpMethod: POST
title: "{{ .CommonLabels.alertname }}"
message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}{{ end }}"
disableResolveMessage: false

View File

@@ -0,0 +1,15 @@
# Grafana alerting provisioning — notification policies
# Routes all alerts to Gotify by default.
apiVersion: 1
policies:
- orgId: 1
receiver: Gotify
group_by: ["alertname", "service"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: Gotify
matchers:
- severity =~ "critical|warning"

View File

@@ -0,0 +1,214 @@
# Grafana alerting provisioning — alert rules
# Covers: runner down, high task failure rate, audio error spike, backend error spike.
apiVersion: 1
groups:
- orgId: 1
name: LibNovel Runner
folder: LibNovel
interval: 1m
rules:
- uid: runner-down
title: Runner Down
condition: C
for: 2m
annotations:
summary: "LibNovel runner is not reachable"
description: "The Prometheus scrape of runner:9091 has been failing for >2 minutes. Tasks are not being processed."
labels:
severity: critical
service: runner
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange: { from: 300, to: 0 }
model:
expr: "up{job=\"libnovel-runner\"}"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 300, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [1], type: lt }
operator: { type: and }
query: { params: [A] }
reducer: { params: [], type: last }
- uid: runner-high-failure-rate
title: Runner High Task Failure Rate
condition: C
for: 5m
annotations:
summary: "Runner task failure rate is above 20%"
description: "More than 20% of runner tasks have been failing for the last 5 minutes. Check runner logs."
labels:
severity: warning
service: runner
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange: { from: 600, to: 0 }
model:
expr: "rate(libnovel_runner_tasks_failed_total[5m]) / clamp_min(rate(libnovel_runner_tasks_completed_total[5m]) + rate(libnovel_runner_tasks_failed_total[5m]), 0.001)"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 600, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [0.2], type: gt }
operator: { type: and }
query: { params: [A] }
reducer: { params: [], type: last }
- uid: runner-tasks-stalled
title: Runner Tasks Stalled
condition: C
for: 10m
annotations:
summary: "Runner has tasks running for >10 minutes with no completions"
description: "tasks_running > 0 but rate(tasks_completed) is 0. Tasks may be stuck or the runner is in a crash loop."
labels:
severity: warning
service: runner
data:
- refId: Running
datasourceUid: prometheus
relativeTimeRange: { from: 900, to: 0 }
model:
expr: "libnovel_runner_tasks_running"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: Rate
datasourceUid: prometheus
relativeTimeRange: { from: 900, to: 0 }
model:
expr: "rate(libnovel_runner_tasks_completed_total[10m])"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 900, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [0], type: gt }
operator: { type: and }
query: { params: [Running] }
reducer: { params: [], type: last }
- evaluator: { params: [0.001], type: lt }
operator: { type: and }
query: { params: [Rate] }
reducer: { params: [], type: last }
- orgId: 1
name: LibNovel Backend
folder: LibNovel
interval: 1m
rules:
- uid: backend-high-error-rate
title: Backend High Error Rate
condition: C
for: 5m
annotations:
summary: "Backend API error rate above 5%"
description: "More than 5% of backend HTTP requests are returning 5xx status codes (as seen from UI OTel instrumentation)."
labels:
severity: warning
service: backend
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange: { from: 600, to: 0 }
model:
expr: "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\", http_response_status_code=~\"5..\"}[5m])) / clamp_min(sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\"}[5m])), 0.001)"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 600, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [0.05], type: gt }
operator: { type: and }
query: { params: [A] }
reducer: { params: [], type: last }
- uid: backend-high-p95-latency
title: Backend High p95 Latency
condition: C
for: 5m
annotations:
summary: "Backend p95 latency above 2s"
description: "95th percentile latency of backend spans has exceeded 2 seconds for >5 minutes."
labels:
severity: warning
service: backend
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange: { from: 600, to: 0 }
model:
expr: "histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 600, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [2], type: gt }
operator: { type: and }
query: { params: [A] }
reducer: { params: [], type: last }
- orgId: 1
name: LibNovel OTel Pipeline
folder: LibNovel
interval: 2m
rules:
- uid: otel-collector-down
title: OTel Collector Down
condition: C
for: 3m
annotations:
summary: "OTel collector is not reachable"
description: "Prometheus cannot scrape otel-collector:8888. Traces and logs may be dropping."
labels:
severity: warning
service: otel-collector
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange: { from: 600, to: 0 }
model:
expr: "up{job=\"otel-collector\"}"
instant: true
intervalMs: 1000
maxDataPoints: 43200
- refId: C
datasourceUid: __expr__
relativeTimeRange: { from: 600, to: 0 }
model:
type: classic_conditions
conditions:
- evaluator: { params: [1], type: lt }
operator: { type: and }
query: { params: [A] }
reducer: { params: [], type: last }

View File

@@ -0,0 +1,338 @@
{
"uid": "libnovel-backend",
"title": "Backend API",
"description": "Request rate, error rate, and latency for the LibNovel backend. Powered by Tempo span metrics and UI OTel instrumentation.",
"tags": ["libnovel", "backend", "api"],
"timezone": "browser",
"refresh": "30s",
"time": { "from": "now-3h", "to": "now" },
"schemaVersion": 39,
"panels": [
{
"id": 1,
"type": "stat",
"title": "Request Rate (RPS)",
"gridPos": { "x": 0, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": { "mode": "thresholds" },
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"backend\"}[5m]))",
"legendFormat": "rps",
"instant": true
}
]
},
{
"id": 2,
"type": "stat",
"title": "Error Rate",
"gridPos": { "x": 4, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "none"
},
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.01 },
{ "color": "red", "value": 0.05 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"backend\", status_code=\"STATUS_CODE_ERROR\"}[5m])) / clamp_min(sum(rate(traces_spanmetrics_calls_total{service=\"backend\"}[5m])), 0.001)",
"legendFormat": "error rate",
"instant": true
}
]
},
{
"id": 3,
"type": "stat",
"title": "p50 Latency",
"gridPos": { "x": 8, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value", "graphMode": "area" },
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.2 },
{ "color": "red", "value": 1 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.50, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p50",
"instant": true
}
]
},
{
"id": 4,
"type": "stat",
"title": "p95 Latency",
"gridPos": { "x": 12, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value", "graphMode": "area" },
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.5 },
{ "color": "red", "value": 2 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p95",
"instant": true
}
]
},
{
"id": 5,
"type": "stat",
"title": "p99 Latency",
"gridPos": { "x": 16, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value", "graphMode": "area" },
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 5 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.99, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p99",
"instant": true
}
]
},
{
"id": 6,
"type": "stat",
"title": "5xx Errors / min",
"gridPos": { "x": 20, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "background", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 5 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\", http_response_status_code=~\"5..\"}[5m])) * 60",
"legendFormat": "5xx/min",
"instant": true
}
]
},
{
"id": 10,
"type": "timeseries",
"title": "Request Rate by Status",
"gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "reqps", "custom": { "lineWidth": 2, "fillOpacity": 10 } },
"overrides": [
{ "matcher": { "id": "byFrameRefID", "options": "errors" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] }
]
},
"targets": [
{
"refId": "success",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\", http_response_status_code=~\"2..\"}[5m]))",
"legendFormat": "2xx"
},
{
"refId": "notfound",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\", http_response_status_code=~\"4..\"}[5m]))",
"legendFormat": "4xx"
},
{
"refId": "errors",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\", http_response_status_code=~\"5..\"}[5m]))",
"legendFormat": "5xx"
}
]
},
{
"id": 11,
"type": "timeseries",
"title": "Latency Percentiles (backend spans)",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "s", "custom": { "lineWidth": 2, "fillOpacity": 10 } }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.50, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p50"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p95"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.99, sum(rate(traces_spanmetrics_latency_bucket{service=\"backend\"}[5m])) by (le))",
"legendFormat": "p99"
}
]
},
{
"id": 12,
"type": "timeseries",
"title": "Requests / min by HTTP method (UI → Backend)",
"gridPos": { "x": 0, "y": 12, "w": 12, "h": 8 },
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "short", "custom": { "lineWidth": 2, "fillOpacity": 5 } }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"backend\"}[5m])) by (http_request_method) * 60",
"legendFormat": "{{http_request_method}}"
}
]
},
{
"id": 13,
"type": "timeseries",
"title": "Requests / min — UI → PocketBase",
"gridPos": { "x": 12, "y": 12, "w": 12, "h": 8 },
"description": "Traffic from SvelteKit server to PocketBase (auth, collections, etc.).",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "short", "custom": { "lineWidth": 2, "fillOpacity": 5 } }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(http_client_request_duration_seconds_count{job=\"ui\", server_address=\"pocketbase\"}[5m])) by (http_request_method, http_response_status_code) * 60",
"legendFormat": "{{http_request_method}} {{http_response_status_code}}"
}
]
},
{
"id": 14,
"type": "timeseries",
"title": "UI → Backend Latency (p50 / p95)",
"gridPos": { "x": 0, "y": 20, "w": 12, "h": 8 },
"description": "HTTP client latency as seen from the SvelteKit SSR layer calling backend.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "s", "custom": { "lineWidth": 2, "fillOpacity": 5 } }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.50, sum(rate(http_client_request_duration_seconds_bucket{job=\"ui\", server_address=\"backend\"}[5m])) by (le))",
"legendFormat": "p50"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.95, sum(rate(http_client_request_duration_seconds_bucket{job=\"ui\", server_address=\"backend\"}[5m])) by (le))",
"legendFormat": "p95"
}
]
},
{
"id": 20,
"type": "logs",
"title": "Backend Errors",
"gridPos": { "x": 0, "y": 28, "w": 24, "h": 10 },
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"backend\"} | json | level =~ `(WARN|ERROR|error|warn)`",
"legendFormat": ""
}
]
}
]
}

View File

@@ -0,0 +1,275 @@
{
"uid": "libnovel-catalogue",
"title": "Catalogue & Content Progress",
"description": "Scraping progress, audio generation coverage, and catalogue health derived from runner structured logs.",
"tags": ["libnovel", "catalogue", "content"],
"timezone": "browser",
"refresh": "1m",
"time": { "from": "now-24h", "to": "now" },
"schemaVersion": 39,
"panels": [
{
"id": 1,
"type": "stat",
"title": "Books Scraped (last 24h)",
"description": "Count of unique book slugs appearing in successful scrape task completions.",
"gridPos": { "x": 0, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "value", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"color": { "fixedColor": "blue", "mode": "fixed" },
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "sum_over_time({service_name=\"runner\"} | json | msg=`scrape task done` [24h])",
"legendFormat": "books scraped"
}
]
},
{
"id": 2,
"type": "stat",
"title": "Chapters Scraped (last 24h)",
"gridPos": { "x": 4, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "value", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"color": { "fixedColor": "blue", "mode": "fixed" },
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "sum_over_time({service_name=\"runner\"} | json | unwrap scraped [24h])",
"legendFormat": "chapters scraped"
}
]
},
{
"id": 3,
"type": "stat",
"title": "Audio Jobs Completed (last 24h)",
"gridPos": { "x": 8, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "value", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"color": { "fixedColor": "green", "mode": "fixed" },
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "sum_over_time({service_name=\"runner\"} | json | msg=`audio task done` [24h])",
"legendFormat": "audio done"
}
]
},
{
"id": 4,
"type": "stat",
"title": "Audio Jobs Failed (last 24h)",
"gridPos": { "x": 12, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 5 }
]
}
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "sum_over_time({service_name=\"runner\"} | json | msg=`audio task failed` [24h])",
"legendFormat": "audio failed"
}
]
},
{
"id": 5,
"type": "stat",
"title": "Scrape Errors (last 24h)",
"gridPos": { "x": 16, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 10 }
]
}
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "sum_over_time({service_name=\"runner\"} | json | msg=`scrape task failed` [24h])",
"legendFormat": "scrape errors"
}
]
},
{
"id": 6,
"type": "stat",
"title": "Catalogue Refresh — Books Indexed",
"description": "Total books indexed in the last catalogue refresh cycle (from the ok field in the summary log).",
"gridPos": { "x": 20, "y": 0, "w": 4, "h": 4 },
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value", "graphMode": "none" },
"fieldConfig": {
"defaults": {
"color": { "fixedColor": "purple", "mode": "fixed" },
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "last_over_time({service_name=\"runner\"} | json | op=`catalogue_refresh` | msg=`catalogue refresh done` | unwrap ok [7d])",
"legendFormat": "indexed"
}
]
},
{
"id": 10,
"type": "timeseries",
"title": "Audio Generation Rate (tasks/min)",
"gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
"description": "Rate of audio task completions and failures over time.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "short", "custom": { "lineWidth": 2, "fillOpacity": 10 } },
"overrides": [
{ "matcher": { "id": "byName", "options": "failed" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "completed" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] }
]
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.audio_task\", status_code!=\"STATUS_CODE_ERROR\"}[5m])) * 60",
"legendFormat": "completed"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.audio_task\", status_code=\"STATUS_CODE_ERROR\"}[5m])) * 60",
"legendFormat": "failed"
}
]
},
{
"id": 11,
"type": "timeseries",
"title": "Scraping Rate (tasks/min)",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"description": "Rate of scrape task completions and failures over time.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": { "unit": "short", "custom": { "lineWidth": 2, "fillOpacity": 10 } },
"overrides": [
{ "matcher": { "id": "byName", "options": "failed" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "completed" }, "properties": [{ "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }] }
]
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.scrape_task\", status_code!=\"STATUS_CODE_ERROR\"}[5m])) * 60",
"legendFormat": "completed"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.scrape_task\", status_code=\"STATUS_CODE_ERROR\"}[5m])) * 60",
"legendFormat": "failed"
}
]
},
{
"id": 20,
"type": "logs",
"title": "Scrape Task Events",
"description": "One log line per completed or failed scrape task. Fields: task_id, kind, url, scraped, skipped, errors.",
"gridPos": { "x": 0, "y": 12, "w": 24, "h": 10 },
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": false,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"runner\"} | json | msg =~ `scrape task (done|failed|starting)`",
"legendFormat": ""
}
]
},
{
"id": 21,
"type": "logs",
"title": "Audio Task Events",
"description": "One log line per completed or failed audio task. Fields: task_id, slug, chapter, voice, key (on success), reason (on failure).",
"gridPos": { "x": 0, "y": 22, "w": 24, "h": 10 },
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": false,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"runner\"} | json | msg =~ `audio task (done|failed|starting)`",
"legendFormat": ""
}
]
},
{
"id": 22,
"type": "logs",
"title": "Catalogue Refresh Progress",
"description": "Progress logs from the background catalogue refresh (every 24h). Fields: op=catalogue_refresh, scraped, ok, skipped, errors.",
"gridPos": { "x": 0, "y": 32, "w": 24, "h": 8 },
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": false,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"runner\"} | json | op=`catalogue_refresh`",
"legendFormat": ""
}
]
}
]
}

View File

@@ -0,0 +1,377 @@
{
"uid": "libnovel-runner",
"title": "Runner Operations",
"description": "Task queue health, throughput, TTS routing, and live logs for the homelab runner.",
"tags": ["libnovel", "runner"],
"timezone": "browser",
"refresh": "30s",
"time": { "from": "now-3h", "to": "now" },
"schemaVersion": 39,
"panels": [
{
"id": 1,
"type": "stat",
"title": "Tasks Running",
"gridPos": { "x": 0, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 3 }
]
},
"mappings": []
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_tasks_running",
"legendFormat": "running",
"instant": true
}
]
},
{
"id": 2,
"type": "stat",
"title": "Tasks Completed (total)",
"gridPos": { "x": 4, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "area",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"color": { "fixedColor": "green", "mode": "fixed" },
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_tasks_completed_total",
"legendFormat": "completed",
"instant": true
}
]
},
{
"id": 3,
"type": "stat",
"title": "Tasks Failed (total)",
"gridPos": { "x": 8, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 1 },
{ "color": "red", "value": 5 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_tasks_failed_total",
"legendFormat": "failed",
"instant": true
}
]
},
{
"id": 4,
"type": "stat",
"title": "Runner Uptime",
"gridPos": { "x": 12, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 60 },
{ "color": "green", "value": 300 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_uptime_seconds",
"legendFormat": "uptime",
"instant": true
}
]
},
{
"id": 5,
"type": "stat",
"title": "Task Failure Rate",
"gridPos": { "x": 16, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.05 },
{ "color": "red", "value": 0.2 }
]
}
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_tasks_failed_total / clamp_min(libnovel_runner_tasks_completed_total + libnovel_runner_tasks_failed_total, 1)",
"legendFormat": "failure rate",
"instant": true
}
]
},
{
"id": 6,
"type": "stat",
"title": "Runner Alive",
"gridPos": { "x": 20, "y": 0, "w": 4, "h": 4 },
"options": {
"reduceOptions": { "calcs": ["lastNotNull"] },
"colorMode": "background",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"mappings": [
{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }
],
"thresholds": { "mode": "absolute", "steps": [] }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "up{job=\"libnovel-runner\"}",
"legendFormat": "runner",
"instant": true
}
]
},
{
"id": 10,
"type": "timeseries",
"title": "Task Throughput (per minute)",
"gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": { "lineWidth": 2, "fillOpacity": 10 }
},
"overrides": [
{ "matcher": { "id": "byName", "options": "failed" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "completed" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] }
]
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(libnovel_runner_tasks_completed_total[5m]) * 60",
"legendFormat": "completed"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "rate(libnovel_runner_tasks_failed_total[5m]) * 60",
"legendFormat": "failed"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "libnovel_runner_tasks_running",
"legendFormat": "running"
}
]
},
{
"id": 11,
"type": "timeseries",
"title": "Audio Task Span Latency (p50 / p95 / p99)",
"gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
"description": "End-to-end latency of runner.audio_task spans from Tempo span metrics.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": { "lineWidth": 2, "fillOpacity": 10 }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.50, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.audio_task\"}[5m])) by (le))",
"legendFormat": "p50"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.audio_task\"}[5m])) by (le))",
"legendFormat": "p95"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.99, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.audio_task\"}[5m])) by (le))",
"legendFormat": "p99"
}
]
},
{
"id": 20,
"type": "timeseries",
"title": "Scrape Task Span Latency (p50 / p95 / p99)",
"gridPos": { "x": 0, "y": 12, "w": 12, "h": 8 },
"description": "End-to-end latency of runner.scrape_task spans from Tempo span metrics.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": { "lineWidth": 2, "fillOpacity": 10 }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.50, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.scrape_task\"}[5m])) by (le))",
"legendFormat": "p50"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.scrape_task\"}[5m])) by (le))",
"legendFormat": "p95"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "histogram_quantile(0.99, sum(rate(traces_spanmetrics_latency_bucket{service=\"runner\", span_name=\"runner.scrape_task\"}[5m])) by (le))",
"legendFormat": "p99"
}
]
},
{
"id": 21,
"type": "timeseries",
"title": "Audio vs Scrape Task Rate",
"gridPos": { "x": 12, "y": 12, "w": 12, "h": 8 },
"description": "Relative throughput of audio generation vs book scraping.",
"options": {
"tooltip": { "mode": "multi" },
"legend": { "displayMode": "list", "placement": "bottom" }
},
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": { "lineWidth": 2, "fillOpacity": 10 }
}
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.audio_task\"}[5m]))",
"legendFormat": "audio tasks/s"
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"expr": "sum(rate(traces_spanmetrics_calls_total{service=\"runner\", span_name=\"runner.scrape_task\"}[5m]))",
"legendFormat": "scrape tasks/s"
}
]
},
{
"id": 30,
"type": "logs",
"title": "Runner Logs (errors & warnings)",
"gridPos": { "x": 0, "y": 20, "w": 24, "h": 10 },
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"runner\"} | json | level =~ `(WARN|ERROR|error|warn)`",
"legendFormat": ""
}
]
},
{
"id": 31,
"type": "logs",
"title": "Runner Logs (all)",
"gridPos": { "x": 0, "y": 30, "w": 24, "h": 10 },
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending",
"dedupStrategy": "none"
},
"targets": [
{
"datasource": { "type": "loki", "uid": "loki" },
"expr": "{service_name=\"runner\"} | json",
"legendFormat": ""
}
]
}
]
}

View File

@@ -1,7 +1,7 @@
# LibNovel homelab runner
#
# Connects to production PocketBase and MinIO via public subdomains.
# All secrets come from Doppler (project=libnovel, config=prd).
# All secrets come from Doppler (project=libnovel, config=prd_homelab).
# Run with: doppler run -- docker compose up -d
#
# Differs from prod runner:
@@ -11,12 +11,31 @@
# - MEILI_URL → https://search.libnovel.cc (Caddy-proxied)
# - VALKEY_ADDR → unset (not exposed publicly)
# - RUNNER_SKIP_INITIAL_CATALOGUE_REFRESH=true
# - Redis service for Asynq task queue (local to homelab, exposed to prod via Caddy TCP proxy)
services:
redis:
image: redis:7-alpine
restart: unless-stopped
volumes:
- redis_data:/data
command: >
redis-server
--appendonly yes
--requirepass "${REDIS_PASSWORD}"
healthcheck:
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
interval: 10s
timeout: 5s
retries: 5
runner:
image: kalekber/libnovel-runner:latest
restart: unless-stopped
stop_grace_period: 135s
depends_on:
redis:
condition: service_healthy
environment:
# ── PocketBase ──────────────────────────────────────────────────────────
POCKETBASE_URL: "https://pb.libnovel.cc"
@@ -42,6 +61,14 @@ services:
KOKORO_URL: "${KOKORO_URL}"
KOKORO_VOICE: "${KOKORO_VOICE}"
# ── Pocket TTS ──────────────────────────────────────────────────────────
POCKET_TTS_URL: "${POCKET_TTS_URL}"
# ── Asynq / Redis (local service) ───────────────────────────────────────
# The runner connects to the local Redis sidecar.
REDIS_ADDR: "redis:6379"
REDIS_PASSWORD: "${REDIS_PASSWORD}"
# ── Runner tuning ───────────────────────────────────────────────────────
RUNNER_WORKER_ID: "${RUNNER_WORKER_ID}"
RUNNER_POLL_INTERVAL: "${RUNNER_POLL_INTERVAL}"
@@ -60,3 +87,6 @@ services:
interval: 60s
timeout: 5s
retries: 3
volumes:
redis_data:

View File

@@ -51,6 +51,7 @@
import { audioStore } from '$lib/audio.svelte';
import { Button } from '$lib/components/ui/button';
import { cn } from '$lib/utils';
import type { Voice } from '$lib/types';
interface Props {
slug: string;
@@ -63,8 +64,8 @@
nextChapter?: number | null;
/** Full chapter list for the book (number + title). Written into the store. */
chapters?: { number: number; title: string }[];
/** List of available voices from the Kokoro API. */
voices?: string[];
/** List of available voices from the backend. */
voices?: Voice[];
}
let {
@@ -78,6 +79,10 @@
voices = []
}: Props = $props();
// ── Derived: voices grouped by engine ──────────────────────────────────
const kokoroVoices = $derived(voices.filter((v) => v.engine === 'kokoro'));
const pocketVoices = $derived(voices.filter((v) => v.engine === 'pocket-tts'));
// ── Voice selector state ────────────────────────────────────────────────
let showVoicePanel = $state(false);
/** Voice whose sample is currently being fetched or playing. */
@@ -86,10 +91,33 @@
let sampleAudio = $state<HTMLAudioElement | null>(null);
/**
* Human-readable label for a voice ID.
* e.g. "af_bella" → "Bella (US F)" | "bm_george" → "George (UK M)"
* Human-readable label for a voice.
* Kokoro: "af_bella" → "Bella (US F)"
* Pocket-TTS: "alba" → "Alba (EN F)"
* Falls back gracefully if called with a bare string (e.g. from the store default).
*/
function voiceLabel(v: string): string {
function voiceLabel(v: Voice | string): string {
// Handle plain string IDs stored in audioStore.voice
if (typeof v === 'string') {
// Try to match against the voices list
const found = voices.find((x) => x.id === v);
if (found) return voiceLabel(found);
// Bare kokoro ID fallback (legacy / default "af_bella")
return kokoroLabelFromId(v);
}
if (v.engine === 'pocket-tts') {
const langLabel = v.lang.toUpperCase().replace('-', '');
const genderLabel = v.gender.toUpperCase();
const name = v.id.replace(/_/g, ' ').replace(/\b\w/g, (c) => c.toUpperCase());
return `${name} (${langLabel} ${genderLabel})`;
}
// Kokoro
return kokoroLabelFromId(v.id);
}
function kokoroLabelFromId(id: string): string {
const langMap: Record<string, string> = {
af: 'US', am: 'US',
bf: 'UK', bm: 'UK',
@@ -112,9 +140,8 @@
pf: 'F', pm: 'M',
zf: 'F', zm: 'M',
};
const prefix = v.slice(0, 2);
const name = v.slice(3);
// Capitalise and strip legacy v0 prefix.
const prefix = id.slice(0, 2);
const name = id.slice(3);
const displayName = name
.replace(/^v0/, '')
.replace(/^([a-z])/, (c: string) => c.toUpperCase());
@@ -316,23 +343,28 @@
// ── API helpers ────────────────────────────────────────────────────────────
type PresignResult =
| { ready: true; url: string }
| { ready: false; enqueued: boolean }; // enqueued=true → presign already POSTed
async function tryPresign(
targetSlug: string,
targetChapter: number,
targetVoice: string
): Promise<string | null> {
): Promise<PresignResult> {
const params = new URLSearchParams({
slug: targetSlug,
n: String(targetChapter),
voice: targetVoice
});
const res = await fetch(`/api/presign/audio?${params}`);
// 202: TTS was just enqueued by the presign endpoint — audio not ready yet.
// 202: presign endpoint already triggered TTS — skip the POST, go straight to polling.
// 404: legacy fallback (should no longer occur after endpoint change).
if (res.status === 202 || res.status === 404) return null;
if (res.status === 202) return { ready: false, enqueued: true };
if (res.status === 404) return { ready: false, enqueued: false };
if (!res.ok) throw new Error(`presign HTTP ${res.status}`);
const data = (await res.json()) as { url: string };
return data.url;
return { ready: true, url: data.url };
}
type AudioStatusResponse =
@@ -394,50 +426,52 @@
try {
// Fast path: already generated
const url = await tryPresign(slug, nextChapter, voice);
if (url) {
const presignResult = await tryPresign(slug, nextChapter, voice);
if (presignResult.ready) {
stopNextProgress();
audioStore.nextProgress = 100;
audioStore.nextAudioUrl = url;
audioStore.nextAudioUrl = presignResult.url;
audioStore.nextStatus = 'prefetched';
return;
}
// Slow path: trigger Kokoro generation (non-blocking POST), then poll.
const res = await fetch(`/api/audio/${slug}/${nextChapter}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ voice })
});
if (!res.ok) throw new Error(`Prefetch generation failed: HTTP ${res.status}`);
// Slow path: trigger generation (or skip POST if presign already enqueued).
if (!presignResult.enqueued) {
const res = await fetch(`/api/audio/${slug}/${nextChapter}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ voice })
});
if (!res.ok) throw new Error(`Prefetch generation failed: HTTP ${res.status}`);
// Whether the server returned 200 (already cached) or 202 (enqueued),
// always presign — the status endpoint no longer returns a proxy URL.
if (res.status === 200) {
// Body is { status: 'done' } — audio confirmed in MinIO. Presign it.
await res.body?.cancel();
}
// else 202: generation enqueued — fall through to poll.
if (res.status !== 200) {
// 202: poll until done.
const final = await pollAudioStatus(slug, nextChapter, voice);
stopNextProgress();
audioStore.nextProgress = 100;
if (final.status === 'failed') {
throw new Error(`Prefetch failed: ${(final as { error?: string }).error ?? 'unknown'}`);
if (res.status === 200) {
// Body is { status: 'done' } — audio confirmed in MinIO. Presign it.
await res.body?.cancel();
stopNextProgress();
audioStore.nextProgress = 100;
const doneUrl = await tryPresign(slug, nextChapter, voice);
if (!doneUrl.ready) throw new Error('Prefetch: audio done but presign returned 404');
audioStore.nextAudioUrl = doneUrl.url;
audioStore.nextStatus = 'prefetched';
return;
}
} else {
stopNextProgress();
audioStore.nextProgress = 100;
// 202: generation enqueued — fall through to poll.
}
// Poll until done (covers both: presign-enqueued and POST-enqueued paths).
const final = await pollAudioStatus(slug, nextChapter, voice);
stopNextProgress();
audioStore.nextProgress = 100;
if (final.status === 'failed') {
throw new Error(`Prefetch failed: ${(final as { error?: string }).error ?? 'unknown'}`);
}
// Audio is ready in MinIO — get a direct presigned URL.
const doneUrl = await tryPresign(slug, nextChapter, voice);
if (!doneUrl) throw new Error('Prefetch: audio done but presign returned 404');
if (!doneUrl.ready) throw new Error('Prefetch: audio done but presign returned 404');
audioStore.nextAudioUrl = doneUrl;
audioStore.nextAudioUrl = doneUrl.url;
audioStore.nextStatus = 'prefetched';
} catch {
stopNextProgress();
@@ -505,9 +539,9 @@
}
// Fast path B: audio already in MinIO (presign check).
const url = await tryPresign(slug, chapter, voice);
if (url) {
audioStore.audioUrl = url;
const presignResult = await tryPresign(slug, chapter, voice);
if (presignResult.ready) {
audioStore.audioUrl = presignResult.url;
audioStore.status = 'ready';
// Restore last saved position after the audio element loads
restoreSavedAudioTime();
@@ -520,33 +554,44 @@
audioStore.status = 'generating';
startProgress();
const res = await fetch(`/api/audio/${slug}/${chapter}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ voice })
});
if (!res.ok) throw new Error(`Generation failed: HTTP ${res.status}`);
// presignResult.enqueued=true means /api/presign/audio already POSTed on our
// behalf — skip the duplicate POST and go straight to polling.
if (!presignResult.enqueued) {
const res = await fetch(`/api/audio/${slug}/${chapter}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ voice })
});
if (!res.ok) throw new Error(`Generation failed: HTTP ${res.status}`);
if (res.status !== 200) {
// 202: generation enqueued — poll until done.
const final = await pollAudioStatus(slug, chapter, voice);
if (final.status === 'failed') {
throw new Error(
`Generation failed: ${(final as { error?: string }).error ?? 'unknown error'}`
);
if (res.status === 200) {
// Already cached — body is { status: 'done' }, no url needed.
await res.body?.cancel();
await finishProgress();
const doneUrl = await tryPresign(slug, chapter, voice);
if (!doneUrl.ready) throw new Error('Audio generated but presign returned 404');
audioStore.audioUrl = doneUrl.url;
audioStore.status = 'ready';
maybeStartPrefetch();
return;
}
} else {
// 200: already cached — body is { status: 'done' }, no url needed.
await res.body?.cancel();
// 202: fall through to polling below.
}
// Poll until the runner finishes generating.
const final = await pollAudioStatus(slug, chapter, voice);
if (final.status === 'failed') {
throw new Error(
`Generation failed: ${(final as { error?: string }).error ?? 'unknown error'}`
);
}
await finishProgress();
// Audio is ready in MinIO — always use a presigned URL for direct playback.
const doneUrl = await tryPresign(slug, chapter, voice);
if (!doneUrl) throw new Error('Audio generated but presign returned 404');
audioStore.audioUrl = doneUrl;
if (!doneUrl.ready) throw new Error('Audio generated but presign returned 404');
audioStore.audioUrl = doneUrl.url;
audioStore.status = 'ready';
// Don't restore time for freshly generated audio — position is 0
// Immediately start pre-generating the next chapter in background.
@@ -627,6 +672,52 @@
<svelte:window onkeydown={handleKeyDown} />
<!-- ── Voice row snippet (reused in both engine sections) ──────────────── -->
{#snippet voiceRow(v: import('$lib/types').Voice)}
<div
class={cn('flex items-center gap-2 px-3 py-2 hover:bg-zinc-800 transition-colors cursor-pointer', audioStore.voice === v.id && 'bg-amber-400/10')}
role="button"
tabindex="0"
onclick={() => selectVoice(v.id)}
onkeydown={(e) => e.key === 'Enter' && selectVoice(v.id)}
>
<!-- Selected indicator -->
<div class="w-4 flex-shrink-0">
{#if audioStore.voice === v.id}
<svg class="w-3.5 h-3.5 text-amber-400" fill="currentColor" viewBox="0 0 24 24">
<path d="M9 16.17L4.83 12l-1.42 1.41L9 19 21 7l-1.41-1.41L9 16.17z"/>
</svg>
{/if}
</div>
<!-- Voice name -->
<span class={cn('flex-1 text-xs', audioStore.voice === v.id ? 'text-amber-400 font-medium' : 'text-zinc-300')}>
{voiceLabel(v)}
</span>
<span class="text-zinc-600 text-xs font-mono">{v.id}</span>
<!-- Sample play button -->
<Button
variant="ghost"
size="icon"
class={cn('h-6 w-6 flex-shrink-0', samplePlayingVoice === v.id ? 'text-amber-400 bg-amber-400/15 hover:bg-amber-400/25' : 'text-zinc-500 hover:text-zinc-200')}
onclick={(e) => { e.stopPropagation(); playSample(v.id); }}
title={samplePlayingVoice === v.id ? 'Stop sample' : 'Play sample'}
aria-label={samplePlayingVoice === v.id ? `Stop ${v.id} sample` : `Play ${v.id} sample`}
>
{#if samplePlayingVoice === v.id}
<svg class="w-3.5 h-3.5" fill="currentColor" viewBox="0 0 24 24">
<path d="M6 6h12v12H6z"/>
</svg>
{:else}
<svg class="w-3.5 h-3.5" fill="currentColor" viewBox="0 0 24 24">
<path d="M8 5v14l11-7z"/>
</svg>
{/if}
</Button>
</div>
{/snippet}
<div class="mt-6 p-4 rounded-lg bg-zinc-800 border border-zinc-700">
<div class="flex items-center justify-between gap-2 mb-3">
<div class="flex items-center gap-2">
@@ -674,50 +765,25 @@
</Button>
</div>
<div class="max-h-64 overflow-y-auto">
{#each voices as v (v)}
<div
class={cn('flex items-center gap-2 px-3 py-2 hover:bg-zinc-800 transition-colors cursor-pointer', audioStore.voice === v && 'bg-amber-400/10')}
role="button"
tabindex="0"
onclick={() => selectVoice(v)}
onkeydown={(e) => e.key === 'Enter' && selectVoice(v)}
>
<!-- Selected indicator -->
<div class="w-4 flex-shrink-0">
{#if audioStore.voice === v}
<svg class="w-3.5 h-3.5 text-amber-400" fill="currentColor" viewBox="0 0 24 24">
<path d="M9 16.17L4.83 12l-1.42 1.41L9 19 21 7l-1.41-1.41L9 16.17z"/>
</svg>
{/if}
</div>
<!-- Voice name -->
<span class={cn('flex-1 text-xs', audioStore.voice === v ? 'text-amber-400 font-medium' : 'text-zinc-300')}>
{voiceLabel(v)}
</span>
<span class="text-zinc-600 text-xs font-mono">{v}</span>
<!-- Sample play button (stop propagation so click doesn't select) -->
<Button
variant="ghost"
size="icon"
class={cn('h-6 w-6 flex-shrink-0', samplePlayingVoice === v ? 'text-amber-400 bg-amber-400/15 hover:bg-amber-400/25' : 'text-zinc-500 hover:text-zinc-200')}
onclick={(e) => { e.stopPropagation(); playSample(v); }}
title={samplePlayingVoice === v ? 'Stop sample' : 'Play sample'}
aria-label={samplePlayingVoice === v ? `Stop ${v} sample` : `Play ${v} sample`}
>
{#if samplePlayingVoice === v}
<svg class="w-3.5 h-3.5" fill="currentColor" viewBox="0 0 24 24">
<path d="M6 6h12v12H6z"/>
</svg>
{:else}
<svg class="w-3.5 h-3.5" fill="currentColor" viewBox="0 0 24 24">
<path d="M8 5v14l11-7z"/>
</svg>
{/if}
</Button>
<!-- Kokoro (GPU) section -->
{#if kokoroVoices.length > 0}
<div class="px-3 py-1.5 bg-zinc-800/70 border-b border-zinc-700/50">
<span class="text-[10px] font-semibold text-zinc-500 uppercase tracking-widest">Kokoro (GPU)</span>
</div>
{/each}
{#each kokoroVoices as v (v.id)}
{@render voiceRow(v)}
{/each}
{/if}
<!-- Pocket TTS (CPU) section -->
{#if pocketVoices.length > 0}
<div class="px-3 py-1.5 bg-zinc-800/70 border-b border-zinc-700/50 {kokoroVoices.length > 0 ? 'border-t border-zinc-700' : ''}">
<span class="text-[10px] font-semibold text-zinc-500 uppercase tracking-widest">Pocket TTS (CPU)</span>
</div>
{#each pocketVoices as v (v.id)}
{@render voiceRow(v)}
{/each}
{/if}
</div>
<div class="px-3 py-2 border-t border-zinc-700 bg-zinc-800/50">
<p class="text-xs text-zinc-500">

View File

@@ -40,8 +40,8 @@ export async function presignAvatarUploadUrl(userId: string, mimeType: string):
}
/**
* Returns a presigned GET URL for a user's avatar, rewritten to the public URL.
* Returns null if no avatar exists.
* Returns a presigned GET URL for a user's avatar from MinIO.
* Returns null if no avatar object exists in MinIO for this user.
*/
export async function presignAvatarUrl(userId: string): Promise<string | null> {
const res = await backendFetch(`/api/presign/avatar/${encodeURIComponent(userId)}`);
@@ -54,6 +54,42 @@ export async function presignAvatarUrl(userId: string): Promise<string | null> {
return data.url ? rewriteHost(data.url) : null;
}
/**
* Resolves the best available avatar URL for a user.
*
* Priority:
* 1. MinIO — if the user has uploaded a custom avatar it will be found here
* (presigned, short-lived GET URL).
* 2. OAuth provider URL — stored in avatar_url when the account was created
* via Google / GitHub OAuth (e.g. https://lh3.googleusercontent.com/...).
* Returned as-is; the browser fetches it directly.
*
* Pass the raw `avatar_url` field from the PocketBase record as `storedValue`
* so this function can distinguish between a MinIO key and a remote URL without
* an extra DB round-trip.
*
* Returns null when neither source yields an avatar.
*/
export async function resolveAvatarUrl(
userId: string,
storedValue: string | null | undefined
): Promise<string | null> {
// 1. Try MinIO first (custom upload takes priority over OAuth picture).
try {
const minioUrl = await presignAvatarUrl(userId);
if (minioUrl) return minioUrl;
} catch {
// MinIO unavailable — fall through to OAuth fallback.
}
// 2. Fall back to OAuth-provided picture URL if it looks like a remote URL.
if (storedValue && storedValue.startsWith('http')) {
return storedValue;
}
return null;
}
/**
* Rewrites the MinIO host in a presigned URL to the public-facing URL.
*

View File

@@ -541,6 +541,19 @@ export async function getUserByUsername(username: string): Promise<User | null>
return listOne<User>('app_users', `username="${username.replace(/"/g, '\\"')}"`);
}
/**
* Look up a user by their PocketBase record ID. Returns null if not found.
*/
export async function getUserById(id: string): Promise<User | null> {
const token = await getToken();
const res = await fetch(`${PB_URL}/api/collections/app_users/records/${encodeURIComponent(id)}`, {
headers: { Authorization: `Bearer ${token}` }
});
if (res.status === 404) return null;
if (!res.ok) return null;
return res.json() as Promise<User>;
}
/**
* Look up a user by email. Returns null if not found.
*/

View File

@@ -6,6 +6,20 @@
* safe to import in both server and client code.
*/
// ── Voice ─────────────────────────────────────────────────────────────────────
/** A single TTS voice returned by GET /api/voices. */
export interface Voice {
/** Voice identifier passed to TTS clients (e.g. "af_bella", "alba"). */
id: string;
/** TTS engine: "kokoro" | "pocket-tts". */
engine: string;
/** Primary language tag (e.g. "en-us", "en-gb", "en", "es", "fr"). */
lang: string;
/** Gender: "f" | "m". */
gender: string;
}
// ── Comments ─────────────────────────────────────────────────────────────────
export interface BookComment {

View File

@@ -244,7 +244,7 @@
href="/catalogue"
class="hidden sm:block text-sm transition-colors {page.url.pathname.startsWith('/catalogue') ? 'text-zinc-100 font-medium' : 'text-zinc-400 hover:text-zinc-100'}"
>
Discover
Catalogue
</a>
<a
href="https://feedback.libnovel.cc"
@@ -332,7 +332,7 @@
onclick={() => (menuOpen = false)}
class="px-3 py-2.5 rounded-lg text-sm font-medium transition-colors {page.url.pathname.startsWith('/catalogue') ? 'bg-zinc-800 text-zinc-100' : 'text-zinc-400 hover:bg-zinc-800 hover:text-zinc-100'}"
>
Discover
Catalogue
</a>
<a
href="https://feedback.libnovel.cc"
@@ -400,7 +400,7 @@
<!-- Top row: site links -->
<nav class="flex flex-wrap items-center justify-center gap-x-5 gap-y-2">
<a href="/books" class="hover:text-zinc-400 transition-colors">Library</a>
<a href="/catalogue" class="hover:text-zinc-400 transition-colors">Discover</a>
<a href="/catalogue" class="hover:text-zinc-400 transition-colors">Catalogue</a>
<a
href="https://feedback.libnovel.cc"
target="_blank"
@@ -426,16 +426,26 @@
</svg>
</a>
</nav>
<!-- Bottom row: legal links + copyright -->
<div class="flex flex-wrap items-center justify-center gap-x-5 gap-y-2 text-zinc-700">
<a href="/disclaimer" class="hover:text-zinc-500 transition-colors">Disclaimer</a>
<a href="/privacy" class="hover:text-zinc-500 transition-colors">Privacy</a>
<a href="/dmca" class="hover:text-zinc-500 transition-colors">DMCA</a>
<span>&copy; {new Date().getFullYear()} libnovel</span>
{#if env.PUBLIC_BUILD_VERSION && env.PUBLIC_BUILD_VERSION !== 'dev'}
<span class="text-zinc-800">{env.PUBLIC_BUILD_VERSION}+{env.PUBLIC_BUILD_COMMIT?.slice(0, 7)}</span>
<!-- Bottom row: legal links + copyright -->
<div class="flex flex-wrap items-center justify-center gap-x-5 gap-y-2 text-zinc-700">
<a href="/disclaimer" class="hover:text-zinc-500 transition-colors">Disclaimer</a>
<a href="/privacy" class="hover:text-zinc-500 transition-colors">Privacy</a>
<a href="/dmca" class="hover:text-zinc-500 transition-colors">DMCA</a>
<span>&copy; {new Date().getFullYear()} libnovel</span>
</div>
<!-- Build version / commit SHA -->
<div class="text-xs tabular-nums font-mono px-2 py-0.5 rounded bg-zinc-800 border border-zinc-700">
{#if env.PUBLIC_BUILD_VERSION && env.PUBLIC_BUILD_VERSION !== 'dev'}
<span class="text-zinc-300" title="Build version">{env.PUBLIC_BUILD_VERSION}</span>
{#if env.PUBLIC_BUILD_COMMIT && env.PUBLIC_BUILD_COMMIT !== 'unknown'}
<span class="text-zinc-500 select-all" title="Commit SHA"
>+{env.PUBLIC_BUILD_COMMIT.slice(0, 7)}</span
>
{/if}
</div>
{:else}
<span class="text-zinc-400">dev</span>
{/if}
</div>
</div>
</footer>
</div>

View File

@@ -1,6 +1,7 @@
import { json, error } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { getUserByUsername } from '$lib/server/pocketbase';
import { resolveAvatarUrl } from '$lib/server/minio';
/**
* GET /api/auth/me
@@ -13,10 +14,11 @@ export const GET: RequestHandler = async ({ locals }) => {
}
// Fetch full record from PocketBase to get avatar_url
const record = await getUserByUsername(locals.user.username).catch(() => null);
const avatarUrl = await resolveAvatarUrl(locals.user.id, record?.avatar_url).catch(() => null);
return json({
id: locals.user.id,
username: locals.user.username,
role: locals.user.role,
avatar_url: record?.avatar_url ?? null
avatar_url: avatarUrl
});
};

View File

@@ -4,6 +4,7 @@ import type { RequestHandler } from './$types';
import { getBook, listChapterIdx } from '$lib/server/pocketbase';
import { log } from '$lib/server/logger';
import { backendFetch } from '$lib/server/scraper';
import type { Voice } from '$lib/types';
/**
* GET /api/chapter/[slug]/[n]
@@ -48,11 +49,11 @@ export const GET: RequestHandler = async ({ params, url, locals }) => {
? '<p>' + chapterData.text.replace(/\n{2,}/g, '</p><p>').replace(/\n/g, '<br>') + '</p>'
: '';
let voices: string[] = [];
let voices: Voice[] = [];
try {
const vRes = await backendFetch('/api/voices');
if (vRes.ok) {
const d = (await vRes.json()) as { voices: string[] };
const d = (await vRes.json()) as { voices: Voice[] };
voices = d.voices ?? [];
}
} catch {
@@ -85,10 +86,10 @@ export const GET: RequestHandler = async ({ params, url, locals }) => {
const chapterIdx = chapters.find((c) => c.number === n);
if (!chapterIdx) error(404, `Chapter ${n} not found`);
let voices: string[] = [];
let voices: Voice[] = [];
try {
if (voicesRes?.ok) {
const data = (await voicesRes.json()) as { voices: string[] };
const data = (await voicesRes.json()) as { voices: Voice[] };
voices = data.voices ?? [];
}
} catch {

View File

@@ -5,9 +5,10 @@ import {
listReplies,
createComment,
getMyVotes,
getUserById,
type CommentSort
} from '$lib/server/pocketbase';
import { presignAvatarUrl } from '$lib/server/minio';
import { resolveAvatarUrl } from '$lib/server/minio';
import { log } from '$lib/server/logger';
/**
@@ -38,13 +39,15 @@ export const GET: RequestHandler = async ({ params, url, locals }) => {
replies: repliesPerComment[i]
}));
// Batch-resolve avatar presign URLs for all unique user_ids
// Batch-resolve avatar URLs for all unique user_ids
// MinIO first (custom upload), fall back to OAuth provider picture.
const allComments = [...topLevel, ...allReplies];
const uniqueUserIds = [...new Set(allComments.map((c) => c.user_id).filter(Boolean))];
const avatarEntries = await Promise.all(
uniqueUserIds.map(async (userId) => {
try {
const url = await presignAvatarUrl(userId);
const user = await getUserById(userId);
const url = await resolveAvatarUrl(userId, user?.avatar_url);
return [userId, url] as [string, string | null];
} catch {
return [userId, null] as [string, null];

View File

@@ -1,6 +1,6 @@
import { json, error } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { presignAvatarUrl } from '$lib/server/minio';
import { presignAvatarUrl, resolveAvatarUrl } from '$lib/server/minio';
import { updateUserAvatarUrl, getUserByUsername } from '$lib/server/pocketbase';
import { backendFetch } from '$lib/server/scraper';
@@ -63,10 +63,6 @@ export const GET: RequestHandler = async ({ locals }) => {
if (!locals.user) error(401, 'Not authenticated');
const record = await getUserByUsername(locals.user.username).catch(() => null);
if (!record?.avatar_url) {
return json({ avatar_url: null });
}
const avatarUrl = await presignAvatarUrl(locals.user.id);
const avatarUrl = await resolveAvatarUrl(locals.user.id, record?.avatar_url).catch(() => null);
return json({ avatar_url: avatarUrl });
};

View File

@@ -1,7 +1,7 @@
import { json, error } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { getPublicProfile, getSubscription } from '$lib/server/pocketbase';
import { presignAvatarUrl } from '$lib/server/minio';
import { resolveAvatarUrl } from '$lib/server/minio';
import { log } from '$lib/server/logger';
/**
@@ -15,11 +15,9 @@ export const GET: RequestHandler = async ({ params, locals }) => {
const profile = await getPublicProfile(username);
if (!profile) error(404, `User "${username}" not found`);
// Resolve avatar presigned URL if set
// Resolve avatar — MinIO first, fall back to OAuth provider picture
let avatarUrl: string | null = null;
if (profile.avatar_url) {
avatarUrl = await presignAvatarUrl(profile.id).catch(() => null);
}
avatarUrl = await resolveAvatarUrl(profile.id, profile.avatar_url).catch(() => null);
// Is the current logged-in user subscribed?
let isSubscribed = false;

View File

@@ -1,11 +1,12 @@
import { json } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { backendFetch } from '$lib/server/scraper';
import type { Voice } from '$lib/types';
/**
* GET /api/voices
* Proxies the voice list from the backend Kokoro.
* Returns { voices: string[] }
* Proxies the voice list from the backend (Kokoro + pocket-tts).
* Returns { voices: Voice[] }
*/
export const GET: RequestHandler = async () => {
try {
@@ -13,7 +14,7 @@ export const GET: RequestHandler = async () => {
if (!res.ok) {
return json({ voices: [] });
}
const data = (await res.json()) as { voices: string[] };
const data = (await res.json()) as { voices: Voice[] };
return json({ voices: data.voices ?? [] });
} catch {
return json({ voices: [] });

View File

@@ -4,6 +4,7 @@ import type { PageServerLoad } from './$types';
import { getBook, listChapterIdx } from '$lib/server/pocketbase';
import { log } from '$lib/server/logger';
import { backendFetch } from '$lib/server/scraper';
import type { Voice } from '$lib/types';
export const load: PageServerLoad = async ({ params, url, locals }) => {
const { slug } = params;
@@ -43,11 +44,11 @@ export const load: PageServerLoad = async ({ params, url, locals }) => {
: '';
// Fetch voices (non-critical for preview)
let voices: string[] = [];
let voices: Voice[] = [];
try {
const vRes = await backendFetch('/api/voices');
if (vRes.ok) {
const d = (await vRes.json()) as { voices: string[] };
const d = (await vRes.json()) as { voices: Voice[] };
voices = d.voices ?? [];
}
} catch {
@@ -93,11 +94,11 @@ export const load: PageServerLoad = async ({ params, url, locals }) => {
const chapterIdx = chapters.find((c) => c.number === n);
if (!chapterIdx) error(404, `Chapter ${n} not found`);
// Parse voices — fall back to a minimal default list on error
let voices: string[] = [];
// Parse voices — fall back to empty list on error
let voices: Voice[] = [];
try {
if (voicesRes?.ok) {
const data = (await voicesRes.json()) as { voices: string[] };
const data = (await voicesRes.json()) as { voices: Voice[] };
voices = data.voices ?? [];
}
} catch {

View File

@@ -1,5 +1,6 @@
<script lang="ts">
import { enhance } from '$app/forms';
import { goto } from '$app/navigation';
import { navigating } from '$app/state';
import { untrack } from 'svelte';
import type { PageData, ActionData } from './$types';
@@ -7,6 +8,29 @@
let { data, form }: { data: PageData; form: ActionData } = $props();
// ── Local filter state (mirrors URL params) ──────────────────────────────
// These are separate from data.* so we can bind them to selects and keep
// the DOM in sync. They sync back from data whenever a navigation completes.
let filterSort = $state(untrack(() => data.sort));
let filterGenre = $state(untrack(() => data.genre));
let filterStatus = $state(untrack(() => data.status));
// Keep local state in sync whenever SvelteKit re-runs the load (URL changed).
$effect(() => {
filterSort = data.sort;
filterGenre = data.genre;
filterStatus = data.status;
});
function navigateWithFilters(overrides: { sort?: string; genre?: string; status?: string }) {
const params = new URLSearchParams();
params.set('sort', overrides.sort ?? filterSort);
params.set('genre', overrides.genre ?? filterGenre);
params.set('status', overrides.status ?? filterStatus);
params.set('page', '1');
goto(`/catalogue?${params.toString()}`);
}
// Track which novel card is currently being navigated to
let loadingSlug = $state<string | null>(null);
@@ -389,11 +413,12 @@
<select
id="filter-sort"
name="sort"
value={data.sort}
bind:value={filterSort}
onchange={() => navigateWithFilters({ sort: filterSort })}
class="bg-zinc-900 border border-zinc-700 text-zinc-200 text-sm rounded px-3 py-2 focus:outline-none focus:border-amber-400 w-full"
>
{#each sorts as s}
<option value={s.value}>{s.label}</option>
<option value={s.value} selected={s.value === filterSort}>{s.label}</option>
{/each}
</select>
</div>
@@ -403,12 +428,13 @@
<select
id="filter-genre"
name="genre"
value={data.genre}
bind:value={filterGenre}
onchange={() => navigateWithFilters({ genre: filterGenre })}
disabled={isRankView}
class="bg-zinc-900 border border-zinc-700 text-zinc-200 text-sm rounded px-3 py-2 focus:outline-none focus:border-amber-400 disabled:opacity-40 disabled:cursor-not-allowed w-full"
>
{#each genres as g}
<option value={g.value}>{g.label}</option>
<option value={g.value} selected={g.value === filterGenre}>{g.label}</option>
{/each}
</select>
</div>
@@ -418,12 +444,13 @@
<select
id="filter-status"
name="status"
value={data.status}
bind:value={filterStatus}
onchange={() => navigateWithFilters({ status: filterStatus })}
disabled={isRankView}
class="bg-zinc-900 border border-zinc-700 text-zinc-200 text-sm rounded px-3 py-2 focus:outline-none focus:border-amber-400 disabled:opacity-40 disabled:cursor-not-allowed w-full"
>
{#each statuses as st}
<option value={st.value}>{st.label}</option>
<option value={st.value} selected={st.value === filterStatus}>{st.label}</option>
{/each}
</select>
</div>
@@ -437,13 +464,6 @@
<a href="/catalogue" class="px-4 py-2 rounded bg-zinc-700 text-zinc-300 text-sm hover:bg-zinc-600 transition-colors">
Reset
</a>
<button
type="submit"
onclick={() => (filtersOpen = false)}
class="px-4 py-2 rounded bg-amber-400 text-zinc-900 text-sm font-semibold hover:bg-amber-300 transition-colors"
>
Apply
</button>
</div>
</form>
{/if}

View File

@@ -1,7 +1,7 @@
import { fail, redirect } from '@sveltejs/kit';
import type { Actions, PageServerLoad } from './$types';
import { changePassword, listUserSessions, getUserByUsername } from '$lib/server/pocketbase';
import { presignAvatarUrl } from '$lib/server/minio';
import { resolveAvatarUrl } from '$lib/server/minio';
import { log } from '$lib/server/logger';
export const load: PageServerLoad = async ({ locals }) => {
@@ -16,13 +16,11 @@ export const load: PageServerLoad = async ({ locals }) => {
log.warn('profile', 'listUserSessions failed (non-fatal)', { err: String(e) });
}
// Fetch avatar presigned URL if user has one
// Fetch avatar — MinIO first, fall back to OAuth provider picture
let avatarUrl: string | null = null;
try {
const record = await getUserByUsername(locals.user.username);
if (record?.avatar_url) {
avatarUrl = await presignAvatarUrl(locals.user.id);
}
avatarUrl = await resolveAvatarUrl(locals.user.id, record?.avatar_url);
} catch (e) {
log.warn('profile', 'avatar fetch failed (non-fatal)', { err: String(e) });
}

View File

@@ -5,6 +5,7 @@
import type { PageData, ActionData } from './$types';
import { audioStore } from '$lib/audio.svelte';
import { browser } from '$app/environment';
import type { Voice } from '$lib/types';
let { data, form }: { data: PageData; form: ActionData } = $props();
@@ -56,14 +57,18 @@
}
// ── Settings ────────────────────────────────────────────────────────────────
let voices = $state<string[]>([]);
let voices = $state<Voice[]>([]);
let voicesLoaded = $state(false);
// Derived: voices grouped by engine
const kokoroVoices = $derived(voices.filter((v) => v.engine === 'kokoro'));
const pocketVoices = $derived(voices.filter((v) => v.engine === 'pocket-tts'));
// Load voices on mount
$effect(() => {
fetch('/api/voices')
.then((r) => r.json())
.then((d: { voices: string[] }) => {
.then((d: { voices: Voice[] }) => {
voices = d.voices ?? [];
voicesLoaded = true;
})
@@ -276,9 +281,20 @@
bind:value={voice}
class="w-full bg-zinc-700 border border-zinc-600 rounded-lg px-3 py-2 text-zinc-100 text-sm focus:outline-none focus:ring-2 focus:ring-amber-400"
>
{#each voices as v}
<option value={v}>{v}</option>
{/each}
{#if kokoroVoices.length > 0}
<optgroup label="Kokoro (GPU)">
{#each kokoroVoices as v}
<option value={v.id}>{v.id}</option>
{/each}
</optgroup>
{/if}
{#if pocketVoices.length > 0}
<optgroup label="Pocket TTS (CPU)">
{#each pocketVoices as v}
<option value={v.id}>{v.id}</option>
{/each}
</optgroup>
{/if}
</select>
{/if}
</div>

View File

@@ -6,7 +6,7 @@ import {
getUserPublicLibrary,
getUserCurrentlyReading
} from '$lib/server/pocketbase';
import { presignAvatarUrl } from '$lib/server/minio';
import { resolveAvatarUrl } from '$lib/server/minio';
import { log } from '$lib/server/logger';
export const load: PageServerLoad = async ({ params, locals }) => {
@@ -15,11 +15,9 @@ export const load: PageServerLoad = async ({ params, locals }) => {
const profile = await getPublicProfile(username).catch(() => null);
if (!profile) error(404, `User "${username}" not found`);
// Resolve avatar
// Resolve avatar — MinIO first, fall back to OAuth provider picture
let avatarUrl: string | null = null;
if (profile.avatar_url) {
avatarUrl = await presignAvatarUrl(profile.id).catch(() => null);
}
avatarUrl = await resolveAvatarUrl(profile.id, profile.avatar_url).catch(() => null);
// Subscription state for the logged-in visitor
let isSubscribed = false;