Commit Graph

12 Commits

Author SHA1 Message Date
Admin
4e70a2981d fix(pipeline): add redis+libretranslate to homelab, make Asynq enqueue errors non-fatal
Some checks failed
CI / Backend (pull_request) Successful in 35s
CI / UI (push) Failing after 37s
CI / Backend (push) Successful in 53s
CI / UI (pull_request) Failing after 27s
- homelab/docker-compose.yml: add redis:7-alpine service (port 6379 bound to host
  so Caddy TLS proxy on prod can reach it), add libretranslate service, add
  redis_data and libretranslate_data volumes
- asynqqueue/producer.go: Asynq enqueue failures are now logged as warnings instead
  of returned as errors — PB record already exists so runner picks it up via poll
- backend/main.go: pass logger to NewProducer

Root cause: Redis was not reachable at 192.168.0.109:6379 because the redis
container had no host port binding. Caddy TLS proxy terminates TLS but could
not TCP-connect to the backend Redis.
2026-03-31 10:18:13 +05:00
Admin
8d95411139 fix(caddy): add SNI connection_policy to layer4 TLS block and anchor redis.libnovel.cc cert
Some checks failed
CI / Backend (pull_request) Successful in 30s
CI / UI (pull_request) Successful in 46s
Release / Test backend (push) Successful in 32s
CI / Backend (push) Successful in 49s
CI / UI (push) Successful in 57s
Release / Check ui (push) Successful in 31s
Release / Docker / caddy (push) Successful in 1m19s
Release / Docker / runner (push) Failing after 1m11s
Release / Docker / ui (push) Successful in 2m1s
Release / Docker / backend (push) Successful in 5m1s
Release / Gitea Release (push) Has been skipped
Without a connection_policy, Caddy resolved the TLS cert by the Docker
internal IP (172.18.0.5) instead of the hostname, causing TLS handshake
failures on :6380 (rediss:// from prod backend → homelab Redis / Asynq).

Changes:
- Caddyfile: add connection_policy { match { sni redis.libnovel.cc } } to
  the layer4 :6380 tls handler so Caddy picks the correct cert
- Caddyfile: add redis.libnovel.cc virtual-host block (respond 404) to
  force Caddy to obtain and cache a TLS cert for that hostname
- homelab/docker-compose.yml: add REDIS_ADDR, REDIS_PASSWORD,
  LIBRETRANSLATE_URL, LIBRETRANSLATE_API_KEY, and
  RUNNER_MAX_CONCURRENT_TRANSLATION to the runner service for parity with
  homelab/runner/docker-compose.yml
2026-03-31 00:02:01 +05:00
Admin
1b4835daeb fix(homelab): switch Fider SMTP to port 587 + STARTTLS
All checks were successful
CI / Backend (pull_request) Successful in 50s
CI / UI (pull_request) Successful in 1m10s
Port 465 (SMTPS) is blocked on the homelab server. Port 587 with STARTTLS
works. Updated FIDER_SMTP_PORT=587 and FIDER_SMTP_ENABLE_STARTTLS=true in
Doppler prd_homelab, and made EMAIL_SMTP_ENABLE_STARTTLS dynamic so it reads
from Doppler instead of being hardcoded.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 22:16:01 +05:00
Admin
c9c12fc4a8 fix(infra): correct doppler entrypoint for watchtower container
All checks were successful
CI / UI (pull_request) Successful in 39s
CI / Backend (pull_request) Successful in 45s
- Fix binary path: /usr/bin/doppler (not /usr/local/bin)
- Mount /root/.doppler config so the container can auth without DOPPLER_TOKEN env
- Set HOME=/root so doppler locates the mounted config directory
- Add explicit --project/--config flags to override directory-scope lookup
- Production: --project libnovel --config prd
- Homelab: --project libnovel --config prd_homelab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 22:05:33 +05:00
Admin
dd35024d02 chore(infra): run watchtower via doppler for fresh secrets on restart
All checks were successful
CI / UI (pull_request) Successful in 39s
CI / Backend (pull_request) Successful in 47s
Mount the host doppler binary into the watchtower container and use it as
the entrypoint so WATCHTOWER_NOTIFICATION_URL and other secrets are fetched
from Doppler each time the container starts, rather than being baked in at
compose-up time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 21:52:32 +05:00
Admin
f75292f531 fix(homelab): add Google + GitHub OAuth env vars to Fider service
All checks were successful
CI / Backend (pull_request) Successful in 45s
CI / UI (pull_request) Successful in 38s
2026-03-28 19:38:11 +05:00
Admin
4a00d953bb feat(ui): show build version in footer; enable watchtower for caddy
Some checks failed
CI / Check ui (pull_request) Successful in 50s
CI / Docker / ui (pull_request) Failing after 38s
CI / Test backend (pull_request) Successful in 3m31s
CI / Docker / backend (pull_request) Failing after 11s
CI / Docker / runner (pull_request) Failing after 11s
CI / Docker / caddy (pull_request) Successful in 11m0s
- Footer now displays build version tag + short commit SHA (from
  PUBLIC_BUILD_VERSION / PUBLIC_BUILD_COMMIT env vars baked in at
  image build time); falls back to 'dev' in local builds
- docker-compose.yml: add watchtower label to caddy service so it
  auto-updates alongside backend/ui/runner on new image pushes
- homelab/docker-compose.yml: use locally-built kokoro-fastapi:latest
  image (consistent with actual homelab setup)
2026-03-28 15:42:22 +05:00
Admin
b30aa23d64 fix(homelab): pocket-tts uses locally-built image, correct start command and volumes
Some checks failed
CI / Check ui (pull_request) Successful in 35s
CI / Test backend (pull_request) Successful in 39s
CI / Docker / ui (pull_request) Successful in 1m27s
CI / Docker / backend (pull_request) Successful in 1m57s
CI / Docker / runner (pull_request) Failing after 35s
CI / Docker / caddy (pull_request) Successful in 6m14s
- pocket-tts has no prebuilt image on ghcr.io — must be built from source on homelab
- Updated image to pocket-tts:latest (local tag), add command with --host 0.0.0.0
- Add model cache volumes (pocket_tts_cache, hf_cache) so model weights survive restarts
- start_period increased to 120s (first startup downloads weights)
2026-03-27 16:29:36 +05:00
Admin
fea09e3e23 feat(tts): add pocket-tts engine alongside kokoro-fastapi
Some checks failed
Release / Test backend (push) Successful in 36s
Release / Check ui (push) Successful in 50s
Release / Docker / caddy (push) Failing after 45s
CI / Test backend (pull_request) Successful in 39s
CI / Check ui (pull_request) Failing after 11s
CI / Docker / ui (pull_request) Has been skipped
CI / Docker / caddy (pull_request) Failing after 11s
Release / Docker / backend (push) Failing after 48s
Release / Docker / runner (push) Failing after 58s
Release / Upload source maps (push) Failing after 39s
CI / Docker / backend (pull_request) Failing after 34s
CI / Docker / runner (pull_request) Failing after 30s
Release / Docker / ui (push) Successful in 2m42s
Release / Gitea Release (push) Has been skipped
- New backend/internal/pockettts package: POST /tts client for
  kyutai-labs/pocket-tts; streams WAV response and transcodes to MP3
  via ffmpeg so the rest of the pipeline stays format-agnostic
- config.PocketTTS struct + POCKET_TTS_URL env var
- runner.Dependencies.PocketTTS field; runAudioTask routes by voice name:
  pockettts.IsPocketTTSVoice (alba, marius, javert, …) → pocket-tts,
  everything else → kokoro-fastapi
- Dockerfile: runner stage switched from distroless/static to alpine:3.21
  with ffmpeg + ca-certificates installed
- homelab compose: runner gets POCKET_TTS_URL=http://pocket-tts:8000
- Doppler prd + prd_homelab: KOKORO_URL=https://tts.libnovel.cc,
  POCKET_TTS_URL=https://pocket-tts.libnovel.cc
2026-03-27 16:17:13 +05:00
Admin
4831c74acc feat(observability+tts): OTel logs for backend/runner/ui; add kokoro-fastapi (GPU) and pocket-tts (CPU) to homelab
Some checks failed
Release / Check ui (push) Successful in 47s
Release / Docker / caddy (push) Successful in 48s
Release / Test backend (push) Successful in 3m24s
CI / Check ui (pull_request) Successful in 52s
CI / Test backend (pull_request) Successful in 3m6s
CI / Docker / caddy (pull_request) Failing after 38s
Release / Upload source maps (push) Failing after 33s
Release / Docker / ui (push) Failing after 2m41s
Release / Docker / backend (push) Successful in 2m48s
Release / Docker / runner (push) Failing after 1m1s
Release / Gitea Release (push) Has been skipped
CI / Docker / ui (pull_request) Successful in 1m48s
CI / Docker / backend (pull_request) Successful in 1m56s
CI / Docker / runner (pull_request) Successful in 1m45s
- otelsetup.Init now returns a *slog.Logger wired to the OTLP log exporter
  so all slog output is shipped to Loki with embedded trace IDs
- backend and runner both adopt the new OTel-bridged logger
- runner.runScrapeTask and runAudioTask now emit structured OTel spans
- ui/hooks.server.ts adds BatchLogRecordProcessor alongside existing trace exporter
- homelab: add kokoro-fastapi GPU service (ghcr.io/remsky/kokoro-fastapi-gpu)
  using deploy.resources.reservations for NVIDIA GPU, exposed internally on :8880
- homelab: add pocket-tts CPU service (ghcr.io/kyutai-labs/pocket-tts) on :8000
- runner KOKORO_URL hardcoded to http://kokoro-fastapi:8880 (fixes DNS failure
  for the stale kokoro.kalekber.cc hostname)
2026-03-27 16:05:22 +05:00
Admin
ee3ed29316 Add OTel distributed tracing to backend, ui, and runner
Some checks failed
CI / Check ui (pull_request) Failing after 44s
CI / Docker / ui (pull_request) Has been skipped
CI / Test backend (pull_request) Successful in 3m30s
CI / Docker / backend (pull_request) Successful in 2m28s
CI / Docker / caddy (pull_request) Successful in 5m22s
CI / Docker / runner (pull_request) Successful in 1m52s
Go backend:
- Add OTel SDK + otelhttp middleware deps (go.mod)
- New internal/otelsetup package: init OTLP/HTTP TracerProvider from env vars
- cmd/backend/main.go: call otelsetup.Init() after logger + ctx setup
- internal/backend/server.go: wrap mux with otelhttp.NewHandler() before
  sentryhttp, so all HTTP spans are recorded

SvelteKit UI:
- Add @opentelemetry/sdk-node, exporter-trace-otlp-http, resources,
  semantic-conventions
- hooks.server.ts: init NodeSDK when OTEL_EXPORTER_OTLP_ENDPOINT is set;
  graceful shutdown on SIGTERM/SIGINT

Config:
- docker-compose.yml: pass OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME
  to backend, runner, and ui services
- homelab/docker-compose.yml: fix runner OTel endpoint to HTTP port 4318
- Doppler prd: OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.libnovel.cc
- Doppler prd_homelab: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

All services no-op gracefully when the env var is unset (local dev).
2026-03-26 23:00:52 +05:00
Admin
a39f660a37 Migrate tooling to homelab; add OTel observability stack
All checks were successful
CI / Test backend (pull_request) Successful in 34s
CI / Check ui (pull_request) Successful in 33s
CI / Docker / backend (pull_request) Successful in 1m26s
CI / Docker / runner (pull_request) Successful in 1m50s
CI / Docker / ui (pull_request) Successful in 1m3s
CI / Docker / caddy (pull_request) Successful in 5m53s
- Remove GlitchTip, Umami, Fider, Gotify, Uptime Kuma, Dozzle, Watchtower
  from prod docker-compose (now run on homelab)
- Add dozzle-agent on prod (127.0.0.1:7007) for homelab Dozzle to connect to
- Remove corresponding subdomain blocks from Caddyfile (now routed via
  Cloudflare Tunnel from homelab)
- Add homelab/docker-compose.yml: unified homelab stack with all migrated
  tooling services plus full OTel stack (Tempo 2.6.1, Loki, Prometheus,
  OTel Collector, Grafana)
- Add homelab/otel/: Tempo, Loki, Prometheus, OTel Collector configs +
  Grafana provisioning (datasources + dashboards)
- Add homelab/dozzle/users.yml for Dozzle auth
2026-03-26 21:22:43 +05:00