We use homelab/docker-compose.yml (full stack) for the homelab deployment,
not homelab/runner/docker-compose.yml (runner-only subset). Removing the
unused directory to prevent confusion.
- Admin layout: SVG icons, active highlight, divider between nav sections
- Scrape page: status filter pills with counts, text + status combined search
- Audio page: status filter pills, cancel jobs, retry failed jobs, mobile cards for cache tab
- Translation page: status filter pills (incl. cancelled), cancel + retry jobs, mobile cancel/retry cards, i18n for all labels
- AI Jobs page: fix concurrent cancel (Set instead of single slot), per-job cancel errors inline, full mobile card layout, i18n title/heading
- Text-gen page: tagline editable input + copy, warnings copy, i18n title/heading
- Book page: chapter cover Save button, audio monitor link, currentShelf pre-populated from server
- pocketbase.ts: add getBookShelf(), shelf field on UserLibraryEntry
- New API route: POST /api/admin/translation/bulk (proxy for translation retry)
- i18n: 15 new admin_translation_*, admin_ai_jobs_*, admin_text_gen_* keys across all 5 locales
- Add homelab/glitchtip/files_api.py with GzipChunk fallback for sentry-cli 3.x
raw zip uploads; bind-mount into both glitchtip-web and glitchtip-worker
- Add release.yaml prune step to delete all but the 10 newest GlitchTip releases
- Reader page: remove dead code and simplify layout
- Mount glitchtip_uploads named volume to /code/uploads on both
glitchtip-web and glitchtip-worker so chunk blobs written by the
web container are readable by the worker during assembly
- Set MEDIA_ROOT=/code/uploads explicitly on both services so the
Django storage path matches the volume mount and DB blob records
- Add @grafana/faro-web-sdk to UI; wire initializeFaro in hooks.client.ts
gated on PUBLIC_FARO_COLLECTOR_URL (no-op in dev)
- Add Grafana Alloy service (faro.receiver) to homelab compose;
Faro endpoint → alloy:12347 (faro.libnovel.cc via cloudflared)
- Add PUBLIC_FARO_COLLECTOR_URL env var to docker-compose.yml UI service
- Add Web Vitals dashboard (web-vitals.json): LCP/INP/CLS/TTFB/FCP p75
stats + LCP/TTFB time-series + Faro exception logs from Loki
- Fix runner.json: strip libnovel_ prefix from all metric names
- Fix backend.json: replace 5 dead http_client_* panels with
spanmetrics-based equivalents (Request Rate by Span Name + Latency
by Span Name p95)
- Fix OTel collector: add service.telemetry.metrics.address: 0.0.0.0:8888
so Prometheus can scrape collector self-metrics
- Add Grafana link to admin nav external tools; add admin_nav_grafana
message key to all 5 locale files; recompile paraglide
Without com.centurylinklabs.watchtower.enable=true the homelab watchtower
(running with --label-enable) silently skipped the runner container,
leaving it stuck on v2.5.60 while fixes accumulated on newer tags.
OAuth login was silently failing: upsertUserSession() queried the
device_fingerprint column which didn't exist in the user_sessions
collection, PocketBase returned 400, the fallback authSessionId was
never written to the DB, and isSessionRevoked() immediately revoked
the cookie on first load after the OAuth redirect.
- scripts/pb-init-v3.sh: add device_fingerprint text field to the
user_sessions create block (new installs) and add an idempotent
add_field migration line (existing installs)
Audio jobs were stuck pending because the homelab runner was
connecting to its own local Redis instead of the prod VPS Redis.
- homelab/docker-compose.yml: change hardcoded REDIS_ADDR=redis:6379
to ${REDIS_ADDR} so Doppler injects rediss://redis.libnovel.cc:6380
(the Caddy TLS proxy that bridges the homelab runner to prod Redis)
ClaimNextTranslationTask and HeartbeatTask were no-ops in the asynq
Consumer, so translation tasks created in PocketBase were never picked
up by the runner. Translation tasks live in PocketBase (not Redis),
so they must be claimed/heartbeated via the underlying pb consumer.
ReapStaleTasks is also delegated so stale translation tasks get reset.
Also removes the LibreTranslate healthcheck from homelab/runner
docker-compose.yml and relaxes depends_on to service_started — the
healthcheck was blocking runner startup until models loaded (~2 min)
and the models are already pre-downloaded in the volume.
- homelab/docker-compose.yml: add redis:7-alpine service (port 6379 bound to host
so Caddy TLS proxy on prod can reach it), add libretranslate service, add
redis_data and libretranslate_data volumes
- asynqqueue/producer.go: Asynq enqueue failures are now logged as warnings instead
of returned as errors — PB record already exists so runner picks it up via poll
- backend/main.go: pass logger to NewProducer
Root cause: Redis was not reachable at 192.168.0.109:6379 because the redis
container had no host port binding. Caddy TLS proxy terminates TLS but could
not TCP-connect to the backend Redis.
Without a connection_policy, Caddy resolved the TLS cert by the Docker
internal IP (172.18.0.5) instead of the hostname, causing TLS handshake
failures on :6380 (rediss:// from prod backend → homelab Redis / Asynq).
Changes:
- Caddyfile: add connection_policy { match { sni redis.libnovel.cc } } to
the layer4 :6380 tls handler so Caddy picks the correct cert
- Caddyfile: add redis.libnovel.cc virtual-host block (respond 404) to
force Caddy to obtain and cache a TLS cert for that hostname
- homelab/docker-compose.yml: add REDIS_ADDR, REDIS_PASSWORD,
LIBRETRANSLATE_URL, LIBRETRANSLATE_API_KEY, and
RUNNER_MAX_CONCURRENT_TRANSLATION to the runner service for parity with
homelab/runner/docker-compose.yml
Port 465 (SMTPS) is blocked on the homelab server. Port 587 with STARTTLS
works. Updated FIDER_SMTP_PORT=587 and FIDER_SMTP_ENABLE_STARTTLS=true in
Doppler prd_homelab, and made EMAIL_SMTP_ENABLE_STARTTLS dynamic so it reads
from Doppler instead of being hardcoded.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mount the host doppler binary into the watchtower container and use it as
the entrypoint so WATCHTOWER_NOTIFICATION_URL and other secrets are fetched
from Doppler each time the container starts, rather than being baked in at
compose-up time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Footer now displays build version tag + short commit SHA (from
PUBLIC_BUILD_VERSION / PUBLIC_BUILD_COMMIT env vars baked in at
image build time); falls back to 'dev' in local builds
- docker-compose.yml: add watchtower label to caddy service so it
auto-updates alongside backend/ui/runner on new image pushes
- homelab/docker-compose.yml: use locally-built kokoro-fastapi:latest
image (consistent with actual homelab setup)
- pocket-tts has no prebuilt image on ghcr.io — must be built from source on homelab
- Updated image to pocket-tts:latest (local tag), add command with --host 0.0.0.0
- Add model cache volumes (pocket_tts_cache, hf_cache) so model weights survive restarts
- start_period increased to 120s (first startup downloads weights)
- otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter
so all slog output is shipped to Loki with embedded trace IDs
- backend and runner both adopt the new OTel-bridged logger
- runner.runScrapeTask and runAudioTask now emit structured OTel spans
- ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter
- homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu)
using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880
- homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000
- runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure
for the stale kokoro.kalekber.cc hostname)
Go backend:
- Add OTel SDK + otelhttp middleware deps (go.mod)
- New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars
- cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup
- internal/backend/server.go: wrap mux with otelhttp.NewHandler() before
sentryhttp, so all HTTP spans are recorded
SvelteKit UI:
- Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources,
semantic-conventions
- hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set;
graceful shutdown on SIGTERM/SIGINT
Config:
- docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME
to backend, runner, and ui services
- homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318
- Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc
- Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
All services no-op gracefully when the env var is unset (local dev).
- Add search.libnovel.cc Caddy vhost proxying to meilisearch:7700
- Pass MEILI_URL + MEILI_API_KEY from Doppler into homelab runner
- Set GODEBUG=preferIPv4=1 to work around missing IPv6 route on homelab
- Update comments to reflect runner now indexes books into Meilisearch
- Add email/email_verified/verification_token/verification_token_exp fields
to app_users PocketBase schema (pb-init-v3.sh)
- Add SMTP env vars to UI service in docker-compose.yml
- New email.ts: raw TLS SMTP mailer via Node tls module, sendVerificationEmail()
- createUser() now takes email param, stores verification token (24h TTL)
- loginUser() throws 'Email not verified' when email_verified is false
- New /verify-email route: validates token, verifies user, auto-logs in
- Login page: email field in register form, check-inbox state after register
- /api/auth/register (iOS): returns { pending_verification, email } instead of token
- Add pb.libnovel.cc and storage.libnovel.cc Caddy virtual hosts for homelab runner
- Add homelab runner docker-compose and libnovel.sh helper script
