Commit Graph

5 Commits

Author SHA1 Message Date
root
899c504d1f feat(import): move PDF parsing to backend; fix heartbeat/reap for import_tasks
All checks were successful
Release / Test backend (push) Successful in 51s
Release / Check ui (push) Successful in 2m2s
Release / Docker (push) Successful in 7m32s
Release / Gitea Release (push) Successful in 1m0s
- parsePDF function restored in import.go (body was orphaned outside function)
- ParseImportFile() called at upload time with 3-min timeout; chapters stored as JSON in MinIO
- runner.go: prefer ChaptersKey path (read pre-parsed JSON) over BookImport.Import()
- ImportChapterStore interface added; store wired in runner/main.go
- HeartbeatTask and ReapStaleTasks now include import_tasks collection
- parseImportTask now returns ChaptersKey in domain.ImportTask
- asynq_runner.go handleImportTask passes ChaptersKey
- pb-init-v3.sh: chapters_key field added to import_tasks schema
2026-04-09 21:19:43 +05:00
root
e088bc056e feat: add PDF/EPUB import functionality
Some checks failed
Release / Test backend (push) Successful in 43s
Release / Check ui (push) Failing after 35s
Release / Docker (push) Has been skipped
Release / Gitea Release (push) Has been skipped
- Add ImportTask/ImportResult types to domain.go
- Add TypeImportBook to asynqqueue for task routing
- Add CreateImportTask to producer and storage layers
- Add ClaimNextImportTask/FinishImportTask to Consumer interfaces
- Add import task handling to runner (polling + Asynq handler)
- Add BookImporter interface to bookstore for PDF/EPUB parsing
- Add backend API endpoints: POST/GET /api/admin/import
- Add SvelteKit UI at /admin/import with task list
- Add nav link in admin layout

Note: PDF/EPUB parsing is a placeholder - needs external library integration.
2026-04-09 10:01:20 +05:00
Admin
8e611840d1 fix: add 30s timeout to PB HTTP client; halve heartbeat tick interval
All checks were successful
Release / Test backend (push) Successful in 32s
Release / Docker / caddy (push) Successful in 42s
Release / Check ui (push) Successful in 44s
Release / Docker / ui (push) Successful in 2m32s
Release / Docker / backend (push) Successful in 2m49s
Release / Docker / runner (push) Successful in 3m26s
Release / Gitea Release (push) Successful in 15s
- storage/pocketbase.go: replace http.DefaultClient (no timeout) with a
  dedicated pbHTTPClient{Timeout: 30s} so a slow/hung PocketBase cannot
  stall the backend or runner indefinitely
- runner/asynq_runner.go: heartbeat ticker was firing at StaleTaskThreshold
  (2 min) == the Docker healthcheck deadline, so a single missed tick would
  mark the container unhealthy; halved to StaleTaskThreshold/2 (1 min)
2026-04-02 18:49:04 +05:00
Admin
5da880d189 fix(runner): add heartbeat + translation polling to asynq mode
All checks were successful
CI / UI (push) Successful in 57s
CI / Backend (pull_request) Successful in 1m4s
CI / Backend (push) Successful in 1m45s
CI / UI (pull_request) Successful in 51s
Two bugs prevented asynq mode from working correctly on the homelab runner:

1. No healthcheck file: asynq mode never writes /tmp/runner.alive, so
   Docker healthcheck always fails. Added heartbeat goroutine that
   writes the file every StaleTaskThreshold (30s).

2. Translation tasks not dispatched: translation uses ClaimNextTranslationTask
   (PocketBase poll queue), not Redis/asynq. Audio + scrape use asynq mux,
   but translation sits in PocketBase forever. Added pollTranslationTasks()
   goroutine that polls PocketBase on the same PollInterval as the old
   poll() loop.

All Go tests pass (go test ./... in backend/).
2026-03-29 20:22:37 +05:00
Admin
fe1a933fd0 feat(queue): replace PocketBase polling with Asynq + Redis
Some checks failed
CI / Docker / caddy (pull_request) Failing after 50s
CI / Check ui (pull_request) Successful in 1m5s
Release / Check ui (push) Successful in 45s
Release / Test backend (push) Successful in 1m29s
CI / Docker / ui (pull_request) Successful in 1m28s
Release / Docker / backend (push) Successful in 3m14s
CI / Test backend (pull_request) Failing after 42s
CI / Docker / backend (pull_request) Has been skipped
CI / Docker / runner (pull_request) Has been skipped
Release / Docker / caddy (push) Successful in 6m48s
Release / Docker / ui (push) Successful in 2m8s
Release / Docker / runner (push) Successful in 2m51s
Release / Upload source maps (push) Failing after 53s
Release / Gitea Release (push) Has been skipped
Introduce a Redis-backed Asynq task queue so the runner consumes TTS
jobs pushed by the backend instead of polling PocketBase.

- backend/internal/asynqqueue: Producer and Consumer wrappers
- backend/internal/runner: AsynqRunner mux, per-instance Prometheus
  registry (fixes duplicate-collector panic in tests), redisConnOpt
- backend/internal/config: REDIS_ADDR / REDIS_PASSWORD env vars
- backend/cmd/{backend,runner}/main.go: wire Redis when env set; fall
  back to legacy poll mode when unset
- Caddyfile: caddy-l4 TCP proxy for redis.libnovel.cc:6380 → homelab
- caddy/Dockerfile: add --with github.com/mholt/caddy-l4
- docker-compose.yml: Caddy exposes 6380, backend/runner get Redis env
- homelab/runner/docker-compose.yml: Redis sidecar, runner depends_on
- homelab/otel/grafana: Grafana dashboards (backend, catalogue, runner)
  and alerting rules / contact-points provisioning
2026-03-28 14:32:40 +05:00