12 Commits

Author	SHA1	Message	Date
Admin	a771405db8	feat(audio): WAV streaming, bulk audio generation admin endpoints, cancel/resume ... - Add StreamAudioWAV() to pocket-tts and Kokoro clients; pocket-tts streams raw WAV directly (no ffmpeg), Kokoro requests response_format:wav with stream:true - GET /api/audio-stream supports ?format=wav for lower-latency first-byte delivery; WAV cached separately in MinIO as {slug}/{n}/{voice}.wav - Add GET /api/admin/audio/jobs with optional ?slug filter - Add POST /api/admin/audio/bulk {slug, voice, from, to, skip_existing, force} where skip_existing=true (default) resumes interrupted bulk jobs - Add POST /api/admin/audio/cancel-bulk {slug} to cancel all pending/running tasks - Add CancelAudioTasksBySlug to taskqueue.Producer + asynqqueue implementation - Add AudioObjectKeyExt to bookstore.AudioStore for format-aware MinIO keys Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-02 16:19:14 +05:00
Admin	4e70a2981d	fix(pipeline): add redis+libretranslate to homelab, make Asynq enqueue errors non-fatal ... - homelab/docker-compose.yml: add redis:7-alpine service (port 6379 bound to host so Caddy TLS proxy on prod can reach it), add libretranslate service, add redis_data and libretranslate_data volumes - asynqqueue/producer.go: Asynq enqueue failures are now logged as warnings instead of returned as errors — PB record already exists so runner picks it up via poll - backend/main.go: pass logger to NewProducer Root cause: Redis was not reachable at 192.168.0.109:6379 because the redis container had no host port binding. Caddy TLS proxy terminates TLS but could not TCP-connect to the backend Redis.	2026-03-31 10:18:13 +05:00
Admin	4e7f8c6266	feat: streaming audio endpoint with MinIO write-through cache ... Add GET /api/audio-stream/{slug}/{n}?voice= that streams MP3 audio to the client as TTS generates it, while simultaneously uploading to MinIO. On subsequent requests the endpoint redirects to the presigned MinIO URL, skipping generation entirely. - PocketTTS: StreamAudioMP3 pipes live WAV response body through ffmpeg (streaming transcode — no full-buffer wait) - Kokoro: StreamAudioMP3 uses stream:true mode, returning MP3 frames directly without the two-step download-link flow - AudioStore: PutAudioStream added for multipart MinIO upload from reader - WriteTimeout bumped 60s → 15min to accommodate full-chapter streams - X-Accel-Buffering: no header disables Caddy/nginx response buffering	2026-03-30 22:02:36 +05:00
Admin	28ac8d8826	feat(translation): machine translation pipeline + admin bulk enqueue UI ... - LibreTranslate client (chunks on blank lines, ≤4500 chars, 3-goroutine semaphore) - Runner translation task loop (OTel, heartbeat, MinIO storage) - PocketBase translation_jobs collection support (create/claim/finish/list) - Per-chapter language switcher on chapter reader (EN/RU/ID/PT/FR, polls until done) - Admin /admin/translation page: bulk enqueue form + live-polling jobs table - New backend routes: POST /api/translation/{slug}/{n}, GET /api/translation/status, GET /api/translation/{slug}/{n}, GET /api/admin/translation/jobs, POST /api/admin/translation/bulk - ListTranslationTasks added to taskqueue.Reader interface + store impl - All builds and tests pass; svelte-check: 0 errors	2026-03-29 11:32:42 +05:00
Admin	aef9e04419	fix(runner): harden catalogue scrape against 429s; disable sourcemap upload ... - scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) + 500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata also switched to retryGet so a single 429 on a book page is retried rather than aborting the whole refresh. - catalogue_refresh.go: per-book delay is now configurable (RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter applied before every metadata fetch. Only metadata is scraped here — chapters are fetched on-demand, not during catalogue refresh. Progress logged every 50 books instead of 100. - config.go / runner.go / main.go: add CatalogueRequestDelay field wired from RUNNER_CATALOGUE_REQUEST_DELAY env var. - release.yaml: comment out upload-sourcemaps job and remove it from the release needs; GlitchTip auth token needs refreshing after DB wipe.	2026-03-28 16:28:36 +05:00
Admin	fe1a933fd0	feat(queue): replace PocketBase polling with Asynq + Redis ... Introduce a Redis-backed Asynq task queue so the runner consumes TTS jobs pushed by the backend instead of polling PocketBase. - backend/internal/asynqqueue: Producer and Consumer wrappers - backend/internal/runner: AsynqRunner mux, per-instance Prometheus registry (fixes duplicate-collector panic in tests), redisConnOpt - backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars - backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall back to legacy poll mode when unset - Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab - caddy/Dockerfile: add --with github.com/mholt/caddy-l4 - docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env - homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on - homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner) and alerting rules / contact-points provisioning	2026-03-28 14:32:40 +05:00
Admin	fea09e3e23	feat(tts): add pocket-tts engine alongside kokoro-fastapi ... - New backend/internal/pockettts package: POST /tts client for kyutai-labs/pocket-tts; streams WAV response and transcodes to MP3 via ffmpeg so the rest of the pipeline stays format-agnostic - config.PocketTTS struct + POCKET_TTS_URL env var - runner.Dependencies.PocketTTS field; runAudioTask routes by voice name: pockettts.IsPocketTTSVoice (alba, marius, javert, …) → pocket-tts, everything else → kokoro-fastapi - Dockerfile: runner stage switched from distroless/static to alpine:3.21 with ffmpeg + ca-certificates installed - homelab compose: runner gets POCKET_TTS_URL=http://pocket-tts:8000 - Doppler prd + prd_homelab: KOKORO_URL=https://tts.libnovel.cc, POCKET_TTS_URL=https://pocket-tts.libnovel.cc	2026-03-27 16:17:13 +05:00
Admin	4831c74acc	feat(observability+tts): OTel logs for backend/runner/ui; add kokoro-fastapi (GPU) and pocket-tts (CPU) to homelab ... - otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter so all slog output is shipped to Loki with embedded trace IDs - backend and runner both adopt the new OTel-bridged logger - runner.runScrapeTask and runAudioTask now emit structured OTel spans - ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter - homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu) using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880 - homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000 - runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure for the stale kokoro.kalekber.cc hostname)	2026-03-27 16:05:22 +05:00
Admin	ee3ed29316	Add OTel distributed tracing to backend, ui, and runner ... Go backend: - Add OTel SDK + otelhttp middleware deps (go.mod) - New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars - cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup - internal/backend/server.go: wrap mux with otelhttp.NewHandler() before sentryhttp, so all HTTP spans are recorded SvelteKit UI: - Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources, semantic-conventions - hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set; graceful shutdown on SIGTERM/SIGINT Config: - docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME to backend, runner, and ui services - homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318 - Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc - Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 All services no-op gracefully when the env var is unset (local dev).	2026-03-26 23:00:52 +05:00
Admin	59e8cdb19a	chore: migrate to v3, Doppler secrets, clean up legacy code ... - Remove all pre-v3 code: scraper, ui-v2, backend v1, ios v1+v2, legacy CI workflows - Flatten v3/ contents to repo root - Add Doppler secrets management (project=libnovel, config=prd) - Add justfile with doppler run wrappers for all docker compose commands - Strip hardcoded env fallbacks from docker-compose.yml - Add minimal README.md - Clean up .gitignore	2026-03-23 17:21:12 +05:00
Admin	3918bc8dc3	fix: serve browse pages from MinIO cache; fix ReapStaleTasks PocketBase filter ... - Runner fetches 9 browse combos (genre×sort×status) every 6h and stores JSON snapshots in MinIO libnovel-browse bucket (browse_refresh.go) - Backend handleBrowse reads page-1 results from MinIO first; falls back to live novelfire.net fetch; returns empty+cached:false on total failure instead of 502 - Add BrowseStore interface (bookstore.go), MinIO put/get helpers (minio.go), Store methods + compile-time assertion (store.go), BucketBrowse config, wiring in cmd/backend and cmd/runner, docker-compose-new bucket init - Fix ReapStaleTasks: PocketBase datetime fields require heartbeat_at=null (not heartbeat_at="") in filter expressions, and nil (not "") in patch payload — was causing 400 errors on every reap cycle	2026-03-15 21:19:28 +05:00
Admin	5825b859b7	feat: add v2 stack (backend, runner, ui-v2) with release workflow ... - backend/: Go API server and runner binaries with PocketBase + MinIO storage - ui-v2/: SvelteKit frontend rewrite - docker-compose-new.yml: compose file for the v2 stack - .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images - scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs - .env.example: document DOCKER_BUILDKIT=1 for Colima users	2026-03-15 19:32:40 +05:00