20 Commits

Author	SHA1	Message	Date
Admin	65f0425b61	fix(i18n): add _inputs param to admin_nav paraglide inner functions; exclude Go binaries from git ... The admin_nav_* message files generated by paraglide used 0-param inner functions but called them with inputs, causing 50 svelte-check type errors. Add _inputs = {} to each inner locale function to match the call signature. Also adds backend/backend and backend/runner to .gitignore — binaries are built inside Dockerfile and should never be committed.	2026-03-31 10:53:44 +05:00
Admin	4e70a2981d	fix(pipeline): add redis+libretranslate to homelab, make Asynq enqueue errors non-fatal ... - homelab/docker-compose.yml: add redis:7-alpine service (port 6379 bound to host so Caddy TLS proxy on prod can reach it), add libretranslate service, add redis_data and libretranslate_data volumes - asynqqueue/producer.go: Asynq enqueue failures are now logged as warnings instead of returned as errors — PB record already exists so runner picks it up via poll - backend/main.go: pass logger to NewProducer Root cause: Redis was not reachable at 192.168.0.109:6379 because the redis container had no host port binding. Caddy TLS proxy terminates TLS but could not TCP-connect to the backend Redis.	2026-03-31 10:18:13 +05:00
Admin	4e7f8c6266	feat: streaming audio endpoint with MinIO write-through cache ... Add GET /api/audio-stream/{slug}/{n}?voice= that streams MP3 audio to the client as TTS generates it, while simultaneously uploading to MinIO. On subsequent requests the endpoint redirects to the presigned MinIO URL, skipping generation entirely. - PocketTTS: StreamAudioMP3 pipes live WAV response body through ffmpeg (streaming transcode — no full-buffer wait) - Kokoro: StreamAudioMP3 uses stream:true mode, returning MP3 frames directly without the two-step download-link flow - AudioStore: PutAudioStream added for multipart MinIO upload from reader - WriteTimeout bumped 60s → 15min to accommodate full-chapter streams - X-Accel-Buffering: no header disables Caddy/nginx response buffering	2026-03-30 22:02:36 +05:00
Admin	5da880d189	fix(runner): add heartbeat + translation polling to asynq mode ... Two bugs prevented asynq mode from working correctly on the homelab runner: 1. No healthcheck file: asynq mode never writes /tmp/runner.alive, so Docker healthcheck always fails. Added heartbeat goroutine that writes the file every StaleTaskThreshold (30s). 2. Translation tasks not dispatched: translation uses ClaimNextTranslationTask (PocketBase poll queue), not Redis/asynq. Audio + scrape use asynq mux, but translation sits in PocketBase forever. Added pollTranslationTasks() goroutine that polls PocketBase on the same PollInterval as the old poll() loop. All Go tests pass (go test ./... in backend/).	2026-03-29 20:22:37 +05:00
Admin	28ac8d8826	feat(translation): machine translation pipeline + admin bulk enqueue UI ... - LibreTranslate client (chunks on blank lines, ≤4500 chars, 3-goroutine semaphore) - Runner translation task loop (OTel, heartbeat, MinIO storage) - PocketBase translation_jobs collection support (create/claim/finish/list) - Per-chapter language switcher on chapter reader (EN/RU/ID/PT/FR, polls until done) - Admin /admin/translation page: bulk enqueue form + live-polling jobs table - New backend routes: POST /api/translation/{slug}/{n}, GET /api/translation/status, GET /api/translation/{slug}/{n}, GET /api/admin/translation/jobs, POST /api/admin/translation/bulk - ListTranslationTasks added to taskqueue.Reader interface + store impl - All builds and tests pass; svelte-check: 0 errors	2026-03-29 11:32:42 +05:00
Admin	801928aadf	fix(scraper): update status and genres selectors for current novelfire.net HTML ... novelfire.net changed its book page structure. Old selectors produced empty status and null genres for every book, causing all Meilisearch filters to return zero results. Old → new: - status: <span class="status"> → <strong class="ongoing|completed|hiatus"> (text lowercased for consistent index values) - genres: <div class="genres"> <a> → <div class="categories"> <a class="property-item"> (text lowercased for consistent index values) Adds TestParseMetadataSelectors to guard against future regressions.	2026-03-28 22:54:35 +05:00
Admin	e71ddc2f8b	fix(backend): add ffmpeg to backend image for pocket-tts voice sample generation ... handlePresignVoiceSample generates voice samples on demand via pocket-tts, which requires WAV→MP3 transcoding via ffmpeg. The backend was using distroless/static (no ffmpeg) so all pocket-tts preview requests returned 500. Switch backend stage to Alpine + ffmpeg, matching the runner image.	2026-03-28 21:24:59 +05:00
Admin	aef9e04419	fix(runner): harden catalogue scrape against 429s; disable sourcemap upload ... - scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) + 500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata also switched to retryGet so a single 429 on a book page is retried rather than aborting the whole refresh. - catalogue_refresh.go: per-book delay is now configurable (RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter applied before every metadata fetch. Only metadata is scraped here — chapters are fetched on-demand, not during catalogue refresh. Progress logged every 50 books instead of 100. - config.go / runner.go / main.go: add CatalogueRequestDelay field wired from RUNNER_CATALOGUE_REQUEST_DELAY env var. - release.yaml: comment out upload-sourcemaps job and remove it from the release needs; GlitchTip auth token needs refreshing after DB wipe.	2026-03-28 16:28:36 +05:00
Admin	c5c167035d	fix(tts): fix pocket-tts voices missing in UI and 500 on first TTS enqueue ... - Add POCKET_TTS_URL env to backend service in docker-compose.yml so pocket-tts voices appear in the voice selector (Doppler secret existed but the env var was never passed to the container) - Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of single-quoted string, causing the duplicate-task guard to always miss - Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues TTS internally on 404; AudioPlayer now skips the redundant POST and polls directly, eliminating the 500 from the PB unique-key conflict	2026-03-28 16:02:46 +05:00
Admin	fe1a933fd0	feat(queue): replace PocketBase polling with Asynq + Redis ... Introduce a Redis-backed Asynq task queue so the runner consumes TTS jobs pushed by the backend instead of polling PocketBase. - backend/internal/asynqqueue: Producer and Consumer wrappers - backend/internal/runner: AsynqRunner mux, per-instance Prometheus registry (fixes duplicate-collector panic in tests), redisConnOpt - backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars - backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall back to legacy poll mode when unset - Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab - caddy/Dockerfile: add --with github.com/mholt/caddy-l4 - docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env - homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on - homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner) and alerting rules / contact-points provisioning	2026-03-28 14:32:40 +05:00
Admin	98e4a87432	feat(tts): dual-engine voice list (kokoro + pocket-tts) ... Expose all available voices from both TTS engines via the /api/voices endpoint. AudioPlayer and profile voice-selector now group voices by engine and show a labelled optgroup. Voice type carries an engine field so the chapter-reader can route synthesis to the correct backend.	2026-03-28 14:32:06 +05:00
Admin	9c8849c6cd	fix(otel): accept full https:// URL in OTEL_EXPORTER_OTLP_ENDPOINT ... WithEndpoint expects host[:port] with no scheme. When Doppler has https://otel.libnovel.cc the backend was crashing with 'invalid port'. Now strip the scheme and enable TLS when prefix is https://.	2026-03-27 19:38:50 +05:00
Admin	fea09e3e23	feat(tts): add pocket-tts engine alongside kokoro-fastapi ... - New backend/internal/pockettts package: POST /tts client for kyutai-labs/pocket-tts; streams WAV response and transcodes to MP3 via ffmpeg so the rest of the pipeline stays format-agnostic - config.PocketTTS struct + POCKET_TTS_URL env var - runner.Dependencies.PocketTTS field; runAudioTask routes by voice name: pockettts.IsPocketTTSVoice (alba, marius, javert, …) → pocket-tts, everything else → kokoro-fastapi - Dockerfile: runner stage switched from distroless/static to alpine:3.21 with ffmpeg + ca-certificates installed - homelab compose: runner gets POCKET_TTS_URL=http://pocket-tts:8000 - Doppler prd + prd_homelab: KOKORO_URL=https://tts.libnovel.cc, POCKET_TTS_URL=https://pocket-tts.libnovel.cc	2026-03-27 16:17:13 +05:00
Admin	4831c74acc	feat(observability+tts): OTel logs for backend/runner/ui; add kokoro-fastapi (GPU) and pocket-tts (CPU) to homelab ... - otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter so all slog output is shipped to Loki with embedded trace IDs - backend and runner both adopt the new OTel-bridged logger - runner.runScrapeTask and runAudioTask now emit structured OTel spans - ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter - homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu) using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880 - homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000 - runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure for the stale kokoro.kalekber.cc hostname)	2026-03-27 16:05:22 +05:00
Admin	ee3ed29316	Add OTel distributed tracing to backend, ui, and runner ... Go backend: - Add OTel SDK + otelhttp middleware deps (go.mod) - New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars - cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup - internal/backend/server.go: wrap mux with otelhttp.NewHandler() before sentryhttp, so all HTTP spans are recorded SvelteKit UI: - Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources, semantic-conventions - hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set; graceful shutdown on SIGTERM/SIGINT Config: - docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME to backend, runner, and ui services - homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318 - Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc - Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 All services no-op gracefully when the env var is unset (local dev).	2026-03-26 23:00:52 +05:00
Admin	7e99fc6d70	fix(runner): fix audio task infinite loop and semaphore race ... Two bugs caused audio tasks to loop endlessly: 1. claimRecord never set heartbeat_at — newly claimed tasks had heartbeat_at=null, which matched the reaper's stale filter (heartbeat_at=null || heartbeat_at<threshold). Tasks were reaped and reset to pending within seconds of being claimed, before the 30s heartbeat goroutine had a chance to write a timestamp. Fix: set heartbeat_at=now() in claimRecord alongside status=running. 2. Audio semaphore was checked AFTER claiming the task. When the semaphore was full the select/break only broke the inner select, not the for loop — the code fell through and launched an uncapped goroutine that blocked forever on <-audioSem drain. The task also stayed status=running with no heartbeat, feeding bug #1. Fix: pre-acquire a semaphore slot BEFORE claiming the task; release it immediately if the queue is empty or claim fails.	2026-03-25 15:09:52 +05:00
Admin	59e8cdb19a	chore: migrate to v3, Doppler secrets, clean up legacy code ... - Remove all pre-v3 code: scraper, ui-v2, backend v1, ios v1+v2, legacy CI workflows - Flatten v3/ contents to repo root - Add Doppler secrets management (project=libnovel, config=prd) - Add justfile with doppler run wrappers for all docker compose commands - Strip hardcoded env fallbacks from docker-compose.yml - Add minimal README.md - Clean up .gitignore	2026-03-23 17:21:12 +05:00
Admin	5b27d501af	feat(v3): Caddy hardening, Watchtower, scrape fixes, docs diagrams ... - Caddy: custom image with caddy-ratelimit plugin, security headers (X-Frame-Options, HSTS, CSP-adjacent, etc.), per-IP rate limiting on auth/scrape/global zones, static error pages (502/503/504), fix routing to remove /api/scrape/* and /api/chapter-text-preview/* direct-to-backend (were bypassing SvelteKit auth middleware) - docker-compose: Caddy build context + error volume, Watchtower service (label-enable mode, 5 min poll), watchtower labels on backend/runner/ui - Scraper: ScrapeChapterList uses retryGet (9 attempts, Retry-After backoff) to fix 429-induced chapter list failures; upTo param stops pagination early for range scrapes - UI: Browse→Catalogue rename (routes, API, links), admin scrape page Continue/Retry buttons, +error.svelte branded error page, type cleanup (removed dead exports, added BookPreviewMeta/BookPreviewResponse to scraper.ts) - Meilisearch: meta_updated field, sort=update fix, facet distribution - Docs: reorganise into docs/d2/ and docs/mermaid/ subdirectories, update all diagrams to reflect Caddy/Watchtower/routing changes, add api-routing.d2 ownership map with auth-level colour coding, regenerate SVGs	2026-03-22 21:10:38 +05:00
Admin	3918bc8dc3	fix: serve browse pages from MinIO cache; fix ReapStaleTasks PocketBase filter ... - Runner fetches 9 browse combos (genre×sort×status) every 6h and stores JSON snapshots in MinIO libnovel-browse bucket (browse_refresh.go) - Backend handleBrowse reads page-1 results from MinIO first; falls back to live novelfire.net fetch; returns empty+cached:false on total failure instead of 502 - Add BrowseStore interface (bookstore.go), MinIO put/get helpers (minio.go), Store methods + compile-time assertion (store.go), BucketBrowse config, wiring in cmd/backend and cmd/runner, docker-compose-new bucket init - Fix ReapStaleTasks: PocketBase datetime fields require heartbeat_at=null (not heartbeat_at="") in filter expressions, and nil (not "") in patch payload — was causing 400 errors on every reap cycle	2026-03-15 21:19:28 +05:00
Admin	5825b859b7	feat: add v2 stack (backend, runner, ui-v2) with release workflow ... - backend/: Go API server and runner binaries with PocketBase + MinIO storage - ui-v2/: SvelteKit frontend rewrite - docker-compose-new.yml: compose file for the v2 stack - .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images - scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs - .env.example: document DOCKER_BUILDKIT=1 for Colima users	2026-03-15 19:32:40 +05:00