14 Commits

Author	SHA1	Message	Date
Admin	e71ddc2f8b	fix(backend): add ffmpeg to backend image for pocket-tts voice sample generation ... handlePresignVoiceSample generates voice samples on demand via pocket-tts, which requires WAV→MP3 transcoding via ffmpeg. The backend was using distroless/static (no ffmpeg) so all pocket-tts preview requests returned 500. Switch backend stage to Alpine + ffmpeg, matching the runner image.	2026-03-28 21:24:59 +05:00
Admin	aef9e04419	fix(runner): harden catalogue scrape against 429s; disable sourcemap upload ... - scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) + 500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata also switched to retryGet so a single 429 on a book page is retried rather than aborting the whole refresh. - catalogue_refresh.go: per-book delay is now configurable (RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter applied before every metadata fetch. Only metadata is scraped here — chapters are fetched on-demand, not during catalogue refresh. Progress logged every 50 books instead of 100. - config.go / runner.go / main.go: add CatalogueRequestDelay field wired from RUNNER_CATALOGUE_REQUEST_DELAY env var. - release.yaml: comment out upload-sourcemaps job and remove it from the release needs; GlitchTip auth token needs refreshing after DB wipe.	2026-03-28 16:28:36 +05:00
Admin	c5c167035d	fix(tts): fix pocket-tts voices missing in UI and 500 on first TTS enqueue ... - Add POCKET_TTS_URL env to backend service in docker-compose.yml so pocket-tts voices appear in the voice selector (Doppler secret existed but the env var was never passed to the container) - Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of single-quoted string, causing the duplicate-task guard to always miss - Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues TTS internally on 404; AudioPlayer now skips the redundant POST and polls directly, eliminating the 500 from the PB unique-key conflict	2026-03-28 16:02:46 +05:00
Admin	fe1a933fd0	feat(queue): replace PocketBase polling with Asynq + Redis ... Introduce a Redis-backed Asynq task queue so the runner consumes TTS jobs pushed by the backend instead of polling PocketBase. - backend/internal/asynqqueue: Producer and Consumer wrappers - backend/internal/runner: AsynqRunner mux, per-instance Prometheus registry (fixes duplicate-collector panic in tests), redisConnOpt - backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars - backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall back to legacy poll mode when unset - Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab - caddy/Dockerfile: add --with github.com/mholt/caddy-l4 - docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env - homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on - homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner) and alerting rules / contact-points provisioning	2026-03-28 14:32:40 +05:00
Admin	98e4a87432	feat(tts): dual-engine voice list (kokoro + pocket-tts) ... Expose all available voices from both TTS engines via the /api/voices endpoint. AudioPlayer and profile voice-selector now group voices by engine and show a labelled optgroup. Voice type carries an engine field so the chapter-reader can route synthesis to the correct backend.	2026-03-28 14:32:06 +05:00
Admin	9c8849c6cd	fix(otel): accept full https:// URL in OTEL_EXPORTER_OTLP_ENDPOINT ... WithEndpoint expects host[:port] with no scheme. When Doppler has https://otel.libnovel.cc the backend was crashing with 'invalid port'. Now strip the scheme and enable TLS when prefix is https://.	2026-03-27 19:38:50 +05:00
Admin	fea09e3e23	feat(tts): add pocket-tts engine alongside kokoro-fastapi ... - New backend/internal/pockettts package: POST /tts client for kyutai-labs/pocket-tts; streams WAV response and transcodes to MP3 via ffmpeg so the rest of the pipeline stays format-agnostic - config.PocketTTS struct + POCKET_TTS_URL env var - runner.Dependencies.PocketTTS field; runAudioTask routes by voice name: pockettts.IsPocketTTSVoice (alba, marius, javert, …) → pocket-tts, everything else → kokoro-fastapi - Dockerfile: runner stage switched from distroless/static to alpine:3.21 with ffmpeg + ca-certificates installed - homelab compose: runner gets POCKET_TTS_URL=http://pocket-tts:8000 - Doppler prd + prd_homelab: KOKORO_URL=https://tts.libnovel.cc, POCKET_TTS_URL=https://pocket-tts.libnovel.cc	2026-03-27 16:17:13 +05:00
Admin	4831c74acc	feat(observability+tts): OTel logs for backend/runner/ui; add kokoro-fastapi (GPU) and pocket-tts (CPU) to homelab ... - otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter so all slog output is shipped to Loki with embedded trace IDs - backend and runner both adopt the new OTel-bridged logger - runner.runScrapeTask and runAudioTask now emit structured OTel spans - ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter - homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu) using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880 - homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000 - runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure for the stale kokoro.kalekber.cc hostname)	2026-03-27 16:05:22 +05:00
Admin	ee3ed29316	Add OTel distributed tracing to backend, ui, and runner ... Go backend: - Add OTel SDK + otelhttp middleware deps (go.mod) - New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars - cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup - internal/backend/server.go: wrap mux with otelhttp.NewHandler() before sentryhttp, so all HTTP spans are recorded SvelteKit UI: - Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources, semantic-conventions - hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set; graceful shutdown on SIGTERM/SIGINT Config: - docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME to backend, runner, and ui services - homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318 - Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc - Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 All services no-op gracefully when the env var is unset (local dev).	2026-03-26 23:00:52 +05:00
Admin	7e99fc6d70	fix(runner): fix audio task infinite loop and semaphore race ... Two bugs caused audio tasks to loop endlessly: 1. claimRecord never set heartbeat_at — newly claimed tasks had heartbeat_at=null, which matched the reaper's stale filter (heartbeat_at=null || heartbeat_at<threshold). Tasks were reaped and reset to pending within seconds of being claimed, before the 30s heartbeat goroutine had a chance to write a timestamp. Fix: set heartbeat_at=now() in claimRecord alongside status=running. 2. Audio semaphore was checked AFTER claiming the task. When the semaphore was full the select/break only broke the inner select, not the for loop — the code fell through and launched an uncapped goroutine that blocked forever on <-audioSem drain. The task also stayed status=running with no heartbeat, feeding bug #1. Fix: pre-acquire a semaphore slot BEFORE claiming the task; release it immediately if the queue is empty or claim fails.	2026-03-25 15:09:52 +05:00
Admin	59e8cdb19a	chore: migrate to v3, Doppler secrets, clean up legacy code ... - Remove all pre-v3 code: scraper, ui-v2, backend v1, ios v1+v2, legacy CI workflows - Flatten v3/ contents to repo root - Add Doppler secrets management (project=libnovel, config=prd) - Add justfile with doppler run wrappers for all docker compose commands - Strip hardcoded env fallbacks from docker-compose.yml - Add minimal README.md - Clean up .gitignore	2026-03-23 17:21:12 +05:00
Admin	5b27d501af	feat(v3): Caddy hardening, Watchtower, scrape fixes, docs diagrams ... - Caddy: custom image with caddy-ratelimit plugin, security headers (X-Frame-Options, HSTS, CSP-adjacent, etc.), per-IP rate limiting on auth/scrape/global zones, static error pages (502/503/504), fix routing to remove /api/scrape/* and /api/chapter-text-preview/* direct-to-backend (were bypassing SvelteKit auth middleware) - docker-compose: Caddy build context + error volume, Watchtower service (label-enable mode, 5 min poll), watchtower labels on backend/runner/ui - Scraper: ScrapeChapterList uses retryGet (9 attempts, Retry-After backoff) to fix 429-induced chapter list failures; upTo param stops pagination early for range scrapes - UI: Browse→Catalogue rename (routes, API, links), admin scrape page Continue/Retry buttons, +error.svelte branded error page, type cleanup (removed dead exports, added BookPreviewMeta/BookPreviewResponse to scraper.ts) - Meilisearch: meta_updated field, sort=update fix, facet distribution - Docs: reorganise into docs/d2/ and docs/mermaid/ subdirectories, update all diagrams to reflect Caddy/Watchtower/routing changes, add api-routing.d2 ownership map with auth-level colour coding, regenerate SVGs	2026-03-22 21:10:38 +05:00
Admin	3918bc8dc3	fix: serve browse pages from MinIO cache; fix ReapStaleTasks PocketBase filter ... - Runner fetches 9 browse combos (genre×sort×status) every 6h and stores JSON snapshots in MinIO libnovel-browse bucket (browse_refresh.go) - Backend handleBrowse reads page-1 results from MinIO first; falls back to live novelfire.net fetch; returns empty+cached:false on total failure instead of 502 - Add BrowseStore interface (bookstore.go), MinIO put/get helpers (minio.go), Store methods + compile-time assertion (store.go), BucketBrowse config, wiring in cmd/backend and cmd/runner, docker-compose-new bucket init - Fix ReapStaleTasks: PocketBase datetime fields require heartbeat_at=null (not heartbeat_at="") in filter expressions, and nil (not "") in patch payload — was causing 400 errors on every reap cycle	2026-03-15 21:19:28 +05:00
Admin	5825b859b7	feat: add v2 stack (backend, runner, ui-v2) with release workflow ... - backend/: Go API server and runner binaries with PocketBase + MinIO storage - ui-v2/: SvelteKit frontend rewrite - docker-compose-new.yml: compose file for the v2 stack - .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images - scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs - .env.example: document DOCKER_BUILDKIT=1 for Colima users	2026-03-15 19:32:40 +05:00