Commit Graph

12 Commits

Author SHA1 Message Date
Admin
c5c167035d fix(tts): fix pocket-tts voices missing in UI and 500 on first TTS enqueue
Some checks failed
CI / Test backend (pull_request) Successful in 40s
CI / Check ui (pull_request) Successful in 45s
Release / Test backend (push) Successful in 40s
Release / Check ui (push) Successful in 41s
Release / Docker / caddy (push) Successful in 1m8s
CI / Docker / runner (pull_request) Failing after 35s
CI / Docker / caddy (pull_request) Successful in 3m39s
CI / Docker / ui (pull_request) Successful in 1m12s
CI / Docker / backend (pull_request) Successful in 3m24s
Release / Upload source maps (push) Failing after 43s
Release / Docker / runner (push) Successful in 2m49s
Release / Docker / ui (push) Successful in 2m51s
Release / Docker / backend (push) Failing after 7m14s
Release / Gitea Release (push) Has been skipped
- Add POCKET_TTS_URL env to backend service in docker-compose.yml so
  pocket-tts voices appear in the voice selector (Doppler secret existed
  but the env var was never passed to the container)
- Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of
  single-quoted string, causing the duplicate-task guard to always miss
- Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues
  TTS internally on 404; AudioPlayer now skips the redundant POST and
  polls directly, eliminating the 500 from the PB unique-key conflict
2026-03-28 16:02:46 +05:00
Admin
fe1a933fd0 feat(queue): replace PocketBase polling with Asynq + Redis
Some checks failed
CI / Docker / caddy (pull_request) Failing after 50s
CI / Check ui (pull_request) Successful in 1m5s
Release / Check ui (push) Successful in 45s
Release / Test backend (push) Successful in 1m29s
CI / Docker / ui (pull_request) Successful in 1m28s
Release / Docker / backend (push) Successful in 3m14s
CI / Test backend (pull_request) Failing after 42s
CI / Docker / backend (pull_request) Has been skipped
CI / Docker / runner (pull_request) Has been skipped
Release / Docker / caddy (push) Successful in 6m48s
Release / Docker / ui (push) Successful in 2m8s
Release / Docker / runner (push) Successful in 2m51s
Release / Upload source maps (push) Failing after 53s
Release / Gitea Release (push) Has been skipped
Introduce a Redis-backed Asynq task queue so the runner consumes TTS
jobs pushed by the backend instead of polling PocketBase.

- backend/internal/asynqqueue: Producer and Consumer wrappers
- backend/internal/runner: AsynqRunner mux, per-instance Prometheus
  registry (fixes duplicate-collector panic in tests), redisConnOpt
- backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars
- backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall
  back to legacy poll mode when unset
- Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab
- caddy/Dockerfile: add --with github.com/mholt/caddy-l4
- docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env
- homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on
- homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner)
  and alerting rules / contact-points provisioning
2026-03-28 14:32:40 +05:00
Admin
98e4a87432 feat(tts): dual-engine voice list (kokoro + pocket-tts)
Expose all available voices from both TTS engines via the /api/voices
endpoint. AudioPlayer and profile voice-selector now group voices by
engine and show a labelled optgroup. Voice type carries an engine field
so the chapter-reader can route synthesis to the correct backend.
2026-03-28 14:32:06 +05:00
Admin
9c8849c6cd fix(otel): accept full https:// URL in OTEL_EXPORTER_OTLP_ENDPOINT
Some checks failed
Release / Check ui (push) Successful in 39s
Release / Docker / caddy (push) Successful in 1m4s
Release / Test backend (push) Successful in 1m38s
Release / Docker / runner (push) Successful in 39s
Release / Docker / backend (push) Successful in 2m12s
Release / Docker / ui (push) Successful in 1m0s
Release / Upload source maps (push) Failing after 5m19s
Release / Gitea Release (push) Has been skipped
WithEndpoint expects host[:port] with no scheme. When Doppler has
https://otel.libnovel.cc the backend was crashing with 'invalid port'.
Now strip the scheme and enable TLS when prefix is https://.
2026-03-27 19:38:50 +05:00
Admin
fea09e3e23 feat(tts): add pocket-tts engine alongside kokoro-fastapi
Some checks failed
Release / Test backend (push) Successful in 36s
Release / Check ui (push) Successful in 50s
Release / Docker / caddy (push) Failing after 45s
CI / Test backend (pull_request) Successful in 39s
CI / Check ui (pull_request) Failing after 11s
CI / Docker / ui (pull_request) Has been skipped
CI / Docker / caddy (pull_request) Failing after 11s
Release / Docker / backend (push) Failing after 48s
Release / Docker / runner (push) Failing after 58s
Release / Upload source maps (push) Failing after 39s
CI / Docker / backend (pull_request) Failing after 34s
CI / Docker / runner (pull_request) Failing after 30s
Release / Docker / ui (push) Successful in 2m42s
Release / Gitea Release (push) Has been skipped
- New backend/internal/pockettts package: POST /tts client for
  kyutai-labs/pocket-tts; streams WAV response and transcodes to MP3
  via ffmpeg so the rest of the pipeline stays format-agnostic
- config.PocketTTS struct + POCKET_TTS_URL env var
- runner.Dependencies.PocketTTS field; runAudioTask routes by voice name:
  pockettts.IsPocketTTSVoice (alba, marius, javert, …) → pocket-tts,
  everything else → kokoro-fastapi
- Dockerfile: runner stage switched from distroless/static to alpine:3.21
  with ffmpeg + ca-certificates installed
- homelab compose: runner gets POCKET_TTS_URL=http://pocket-tts:8000
- Doppler prd + prd_homelab: KOKORO_URL=https://tts.libnovel.cc,
  POCKET_TTS_URL=https://pocket-tts.libnovel.cc
2026-03-27 16:17:13 +05:00
Admin
4831c74acc feat(observability+tts): OTel logs for backend/runner/ui; add kokoro-fastapi (GPU) and pocket-tts (CPU) to homelab
Some checks failed
Release / Check ui (push) Successful in 47s
Release / Docker / caddy (push) Successful in 48s
Release / Test backend (push) Successful in 3m24s
CI / Check ui (pull_request) Successful in 52s
CI / Test backend (pull_request) Successful in 3m6s
CI / Docker / caddy (pull_request) Failing after 38s
Release / Upload source maps (push) Failing after 33s
Release / Docker / ui (push) Failing after 2m41s
Release / Docker / backend (push) Successful in 2m48s
Release / Docker / runner (push) Failing after 1m1s
Release / Gitea Release (push) Has been skipped
CI / Docker / ui (pull_request) Successful in 1m48s
CI / Docker / backend (pull_request) Successful in 1m56s
CI / Docker / runner (pull_request) Successful in 1m45s
- otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter
  so all slog output is shipped to Loki with embedded trace IDs
- backend and runner both adopt the new OTel-bridged logger
- runner.runScrapeTask and runAudioTask now emit structured OTel spans
- ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter
- homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu)
  using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880
- homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000
- runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure
  for the stale kokoro.kalekber.cc hostname)
2026-03-27 16:05:22 +05:00
Admin
ee3ed29316 Add OTel distributed tracing to backend, ui, and runner
Some checks failed
CI / Check ui (pull_request) Failing after 44s
CI / Docker / ui (pull_request) Has been skipped
CI / Test backend (pull_request) Successful in 3m30s
CI / Docker / backend (pull_request) Successful in 2m28s
CI / Docker / caddy (pull_request) Successful in 5m22s
CI / Docker / runner (pull_request) Successful in 1m52s
Go backend:
- Add OTel SDK + otelhttp middleware deps (go.mod)
- New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars
- cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup
- internal/backend/server.go: wrap mux with otelhttp.NewHandler() before
  sentryhttp, so all HTTP spans are recorded

SvelteKit UI:
- Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources,
  semantic-conventions
- hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set;
  graceful shutdown on SIGTERM/SIGINT

Config:
- docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME
  to backend, runner, and ui services
- homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318
- Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc
- Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

All services no-op gracefully when the env var is unset (local dev).
2026-03-26 23:00:52 +05:00
Admin
7e99fc6d70 fix(runner): fix audio task infinite loop and semaphore race
Some checks failed
Release / Check ui (push) Successful in 22s
Release / Test backend (push) Successful in 33s
Release / Docker / backend (push) Failing after 30s
Release / Docker / caddy (push) Successful in 1m9s
Release / Docker / ui (push) Successful in 1m34s
Release / Docker / runner (push) Failing after 1m15s
Release / Gitea Release (push) Has been skipped
Two bugs caused audio tasks to loop endlessly:

1. claimRecord never set heartbeat_at — newly claimed tasks had
   heartbeat_at=null, which matched the reaper's stale filter
   (heartbeat_at=null || heartbeat_at<threshold). Tasks were reaped
   and reset to pending within seconds of being claimed, before the
   30s heartbeat goroutine had a chance to write a timestamp.
   Fix: set heartbeat_at=now() in claimRecord alongside status=running.

2. Audio semaphore was checked AFTER claiming the task. When the
   semaphore was full the select/break only broke the inner select,
   not the for loop — the code fell through and launched an uncapped
   goroutine that blocked forever on <-audioSem drain. The task also
   stayed status=running with no heartbeat, feeding bug #1.
   Fix: pre-acquire a semaphore slot BEFORE claiming the task; release
   it immediately if the queue is empty or claim fails.
2026-03-25 15:09:52 +05:00
Admin
59e8cdb19a chore: migrate to v3, Doppler secrets, clean up legacy code
Some checks failed
CI / v3 / Check ui (pull_request) Failing after 15s
CI / v3 / Test backend (pull_request) Failing after 16s
CI / v3 / Docker / backend (pull_request) Has been skipped
CI / v3 / Docker / runner (pull_request) Has been skipped
CI / v3 / Docker / ui (pull_request) Has been skipped
- Remove all pre-v3 code: scraper, ui-v2, backend v1, ios v1+v2, legacy CI workflows
- Flatten v3/ contents to repo root
- Add Doppler secrets management (project=libnovel, config=prd)
- Add justfile with doppler run wrappers for all docker compose commands
- Strip hardcoded env fallbacks from docker-compose.yml
- Add minimal README.md
- Clean up .gitignore
2026-03-23 17:21:12 +05:00
Admin
5b27d501af feat(v3): Caddy hardening, Watchtower, scrape fixes, docs diagrams
- Caddy: custom image with caddy-ratelimit plugin, security headers
  (X-Frame-Options, HSTS, CSP-adjacent, etc.), per-IP rate limiting on
  auth/scrape/global zones, static error pages (502/503/504), fix routing
  to remove /api/scrape/* and /api/chapter-text-preview/* direct-to-backend
  (were bypassing SvelteKit auth middleware)
- docker-compose: Caddy build context + error volume, Watchtower service
  (label-enable mode, 5 min poll), watchtower labels on backend/runner/ui
- Scraper: ScrapeChapterList uses retryGet (9 attempts, Retry-After backoff)
  to fix 429-induced chapter list failures; upTo param stops pagination early
  for range scrapes
- UI: Browse→Catalogue rename (routes, API, links), admin scrape page
  Continue/Retry buttons, +error.svelte branded error page, type cleanup
  (removed dead exports, added BookPreviewMeta/BookPreviewResponse to scraper.ts)
- Meilisearch: meta_updated field, sort=update fix, facet distribution
- Docs: reorganise into docs/d2/ and docs/mermaid/ subdirectories, update
  all diagrams to reflect Caddy/Watchtower/routing changes, add
  api-routing.d2 ownership map with auth-level colour coding, regenerate SVGs
2026-03-22 21:10:38 +05:00
Admin
3918bc8dc3 fix: serve browse pages from MinIO cache; fix ReapStaleTasks PocketBase filter
All checks were successful
Release / Scraper / Test (push) Successful in 20s
Release / UI / Build (push) Successful in 25s
CI / Scraper / Lint (pull_request) Successful in 10s
Release / v2 / Build ui-v2 (push) Successful in 28s
CI / Scraper / Test (pull_request) Successful in 15s
CI / UI / Build (pull_request) Successful in 25s
Release / Scraper / Docker (push) Successful in 55s
Release / UI / Docker (push) Successful in 43s
CI / Scraper / Docker Push (pull_request) Has been skipped
CI / UI / Docker Push (pull_request) Has been skipped
Release / v2 / Docker / ui-v2 (push) Successful in 36s
Release / v2 / Test backend (push) Successful in 3m51s
Release / v2 / Docker / runner (push) Successful in 36s
Release / v2 / Docker / backend (push) Successful in 1m34s
iOS CI / Build (pull_request) Successful in 5m33s
iOS CI / Test (pull_request) Successful in 11m25s
- Runner fetches 9 browse combos (genre×sort×status) every 6h and stores
  JSON snapshots in MinIO libnovel-browse bucket (browse_refresh.go)
- Backend handleBrowse reads page-1 results from MinIO first; falls back
  to live novelfire.net fetch; returns empty+cached:false on total failure
  instead of 502
- Add BrowseStore interface (bookstore.go), MinIO put/get helpers (minio.go),
  Store methods + compile-time assertion (store.go), BucketBrowse config,
  wiring in cmd/backend and cmd/runner, docker-compose-new bucket init
- Fix ReapStaleTasks: PocketBase datetime fields require heartbeat_at=null
  (not heartbeat_at="") in filter expressions, and nil (not "") in patch
  payload — was causing 400 errors on every reap cycle
2026-03-15 21:19:28 +05:00
Admin
5825b859b7 feat: add v2 stack (backend, runner, ui-v2) with release workflow
All checks were successful
Release / Scraper / Test (push) Successful in 10s
Release / UI / Build (push) Successful in 26s
Release / v2 / Build ui-v2 (push) Successful in 17s
Release / Scraper / Docker (push) Successful in 47s
Release / UI / Docker (push) Successful in 56s
CI / Scraper / Lint (pull_request) Successful in 7s
CI / Scraper / Test (pull_request) Successful in 8s
CI / UI / Build (pull_request) Successful in 16s
CI / Scraper / Docker Push (pull_request) Has been skipped
CI / UI / Docker Push (pull_request) Has been skipped
Release / v2 / Docker / ui-v2 (push) Successful in 56s
Release / v2 / Test backend (push) Successful in 4m35s
iOS CI / Build (pull_request) Successful in 4m28s
Release / v2 / Docker / backend (push) Successful in 1m29s
Release / v2 / Docker / runner (push) Successful in 1m39s
iOS CI / Test (pull_request) Successful in 9m51s
- backend/: Go API server and runner binaries with PocketBase + MinIO storage
- ui-v2/: SvelteKit frontend rewrite
- docker-compose-new.yml: compose file for the v2 stack
- .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images
- scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs
- .env.example: document DOCKER_BUILDKIT=1 for Colima users
2026-03-15 19:32:40 +05:00