Admins can now generate TTS audiobooks chapter-by-chapter and publish
them as standard RSS 2.0 + iTunes podcast feeds that Spotify, Apple
Podcasts, and any podcast app can subscribe to.
Backend:
- POST /api/admin/podcast — creates ai_job (kind=podcast), spawns
goroutine that generates TTS for missing chapters and writes to MinIO
- GET /podcast/{slug}.xml?voice=<id> — public RSS feed with correct
pubDate (from chapters_idx.created) and enclosure length (MinIO stat)
- GET /podcast/audio/{slug}/{n}/{voice} — public audio proxy, 302 to
presigned MinIO URL (no auth required for podcast clients)
- GET /api/admin/podcast/{slug} — list podcast jobs for a book
- Add AudioObjectSize to AudioStore interface backed by MinIO StatObject
- Populate ChapterInfo.Date from chapters_idx.created in ListChapters
UI:
- New /admin/podcast page: book + voice selector, chapter range, live
progress bars, copy-feed-URL button, cancel button, how-to instructions
- /api/admin/podcast SvelteKit proxy (injects admin Bearer token)
- Podcast link added to admin sidebar
Cleanup:
- Delete stray playwright screenshot files from repo root
- Add .playwright-mcp/ to .gitignore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Content visibility:
- Add `visibility` field to books ("public" | "admin_only"); new migration
backfills all existing scraped books to admin_only
- Meilisearch: add visibility as filterable attribute; catalogue/search
endpoints filter to public-only for non-admin requests
- Admin users identified by bearer token bypass the filter and see all books
- All PocketBase discovery queries (trending, recommended, recently-updated,
audio shelf, discover, subscription feed) now filter to visibility=public
- New scraped books default to admin_only; WriteMetadata preserves existing
visibility on PATCH (never overwrites)
Author submission:
- POST /api/admin/books/submit — creates a public book with submitted_by
- PATCH /api/admin/books/{slug}/publish / unpublish — toggle visibility
- SvelteKit proxies: /api/admin/books/[slug]/publish|unpublish
- /api/books/[slug] endpoint for admin book lookup
Frontend:
- backendFetchAdmin() helper sends admin token on any path
- Catalogue server load uses admin fetch when user is admin
- /submit page: author submission form with genre picker and rights assertion
- "Publish" nav link shown to all logged-in users
- Admin catalogue-tools: visibility management panel (load book by slug, toggle)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds BACKEND_ADMIN_TOKEN env var (set in Doppler) as a required Bearer
token for every admin route. Also fixes PocketBase filter injection in
notification queries and wires BACKEND_ADMIN_TOKEN through docker-compose
to both backend and ui services. Includes CLAUDE.md for AI assistant guidance.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add bookstore.NotificationStore interface and wire it directly to *storage.Store
in Dependencies, bypassing the Asynq wrapper that caused Producer.(*storage.Store)
to fail with a 500 on every notification endpoint when Redis is configured
- Replace s.deps.Producer.(*storage.Store) type assertion in all 5 notification
handlers with s.deps.NotificationStore (nil-safe, always works)
- Fix PocketBase filter single-quote bug in ListNotifications, ClearAllNotifications,
and MarkAllNotificationsRead (PocketBase requires double quotes for string values)
- Replace bell dropdown with full-screen NotificationsModal (mirrors SearchModal pattern)
- Notifications visible to all logged-in users (not just admin)
- Admin users excluded from new-chapter fan-out (dedup vs Scrape Complete notification)
- Users with notify_new_chapters=false opted out of new-chapter in-app notifications
- Toggle in profile page to enable/disable in-app new-chapter notifications
- PATCH /api/profile endpoint to save notification preferences
- User-facing /notifications page (admin redirects to /admin/notifications)
- Service worker (src/service-worker.ts) handles push events and
notification clicks, navigating to the book page on tap
- Web app manifest (manifest.webmanifest) linked in app.html
- Profile page: push notification toggle (subscribe/unsubscribe)
using the browser Notification + PushManager API with VAPID
- API route POST/DELETE /api/push-subscription proxies to backend
- Go backend: push_subscriptions PocketBase collection storage
methods (SavePushSubscription, DeletePushSubscription,
ListPushSubscriptionsByBook) in storage/store.go
- handlers_push.go: GET vapid-public-key, POST/DELETE subscription
- webpush package: VAPID-signed sends via webpush-go, SendToBook
fans out to all users who have the book in their library
- Runner fires push to subscribers whenever ChaptersScraped > 0
after a successful book scrape
- Config: VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY, VAPID_SUBJECT env vars
- domain.ScrapeResult gets a Slug field; orchestrator populates it
- Add `archived` bool to domain.BookMeta, pbBook, and Meilisearch bookDoc
- ArchiveBook / UnarchiveBook patch the PocketBase record; ListBooks filters
archived=false so hidden books disappear from all public responses
- Meilisearch: add `archived` as a filterable attribute; Search and Catalogue
always prepend `archived = false` to exclude archived books from results
- DeleteBook permanently removes the PocketBase record, all chapters_idx rows,
MinIO chapter objects, cover image, and the Meilisearch document
- New BookAdminStore interface with ArchiveBook, UnarchiveBook, DeleteBook
- Admin HTTP endpoints: PATCH /api/admin/books/{slug}/archive|unarchive,
DELETE /api/admin/books/{slug}
- PocketBase schema: archived field added to live pb.libnovel.cc and to
pb-init-v3.sh (both create block and add_field migration)
- parsePDF: return all text as single 'Full Text' chapter (admin splits manually)
- parseEPUB: fix chapter numbering to use sequential counter not spine index
- Remove dead code: chaptersFromBookmarks, cleanChapterText, extractChaptersFromText, chapterHeadingRE; drop pdfcpu alias and regexp imports
- Backend: POST /api/admin/books/:slug/split-chapters endpoint — splits text on '---' dividers, optional '## Title' headers, writes chapters via WriteChapter
- UI: admin panel now shows for all admin users regardless of source_url; chapter split tool shown when book has single 'Full Text' chapter, pre-fills from MinIO content
Three fixes to PDF chapter extraction quality:
1. Page ordering: parse page number from pdfcpu filename (out_Content_page_N.txt)
instead of using lexicographic sort index — fixes chapters bleeding into each
other (e.g. Prologue text appearing inside Chapter 1).
2. Windows-1252 chars: map bytes 0x91-0x9F to proper Unicode (curly quotes U+2018/
U+2019/U+201C/U+201D, em-dash U+2014, etc.) instead of raw Latin-1 control
bytes that rendered as ◆ in the browser.
3. Chapter header cleanup: skip the first page of each bookmark range (decorative
title art page) and strip any run-on title fragment at the start of the first
body page (e.g. 'for New Journeys!I stood atop...' → 'I stood atop...'). The
remaining sentence truncation is a fundamental limitation of this PDF's
PUA-encoded body font (C2_1/Literata) — those glyphs cannot be decoded without
the publisher's private ToUnicode mapping.
Use PDF outline (bookmarks) for chapter titles and page ranges, then
extract text from per-page content streams via pdfcpu ExtractContent.
This avoids the indefinite hang caused by dslipak/pdf trying to resolve
custom font ToUnicode CMaps on publisher PDFs.
- parsePDF: decrypt → ExtractContent → Bookmarks → chaptersFromBookmarks
- chaptersFromBookmarks: flatten bookmark tree, skip front/back matter
(Cover, Insert, Title Page, Copyright, Appendix), assign page ranges
- extractTextFromContentStream: handle TJ arrays (concat literal strings,
skip hex glyph arrays and kerning numbers) + single Tj strings
- Falls back to paragraph-splitting when no bookmarks present
- Build verified; test PDF produces 9 chapters with proper titles
- parsePDF function restored in import.go (body was orphaned outside function)
- ParseImportFile() called at upload time with 3-min timeout; chapters stored as JSON in MinIO
- runner.go: prefer ChaptersKey path (read pre-parsed JSON) over BookImport.Import()
- ImportChapterStore interface added; store wired in runner/main.go
- HeartbeatTask and ReapStaleTasks now include import_tasks collection
- parseImportTask now returns ChaptersKey in domain.ImportTask
- asynq_runner.go handleImportTask passes ChaptersKey
- pb-init-v3.sh: chapters_key field added to import_tasks schema
- parsePDF now attempts to strip encryption via pdfcpu (empty user password)
before handing bytes to dslipak/pdf — fixes '256-bit encryption key' error
on publisher PDFs that use owner-only encryption (copy/print restrictions)
- Add pdfcpu v0.11.1 as direct dependency (was already indirect)
- docker-compose.yml minio-init: add 'imports' and 'translations' buckets
so a fresh deploy creates all required buckets
- Import task: persist object_key, author, cover_url, genres, summary,
book_status in PocketBase so the runner can fetch the file and write
book metadata on completion
- Runner poll mode: pass task.ObjectKey instead of empty string
- Runner: write BookMeta + UpsertBook in Meilisearch after chapter ingest
so imported books appear in catalogue and search
- Import UI: add author, cover URL, genres, summary, status fields; add
AI tasks panel (chapter names, description, image gen, tagline) after
import completes; add AI tasks button on each done task in the list
- Admin nav: add Notifications entry to sidebar (all 5 locales)
- Logout: delete user_sessions row on sign-out so sessions don't
accumulate as phantoms after each login/logout cycle
- docker-compose.yml: BODY_SIZE_LIMIT=52428800 (50MB) on UI service
— adapter-node was rejecting PDFs >512KB with 'Content-length exceeds limit'
- storage/import.go: add AnalyzeFile() public function using real PDF/EPUB parsers
- handlers_import.go: analyzeImportFile() now calls storage.AnalyzeFile() instead
of the file-size stub, so chapter count is accurate in the preview
- NewBookImporter now takes *Store instead of raw *minio.Client (same package access)
- Runner Dependencies gains ChapterIngester interface; storeImportedChapters no-op removed
- store.IngestChapters is now actually called after PDF/EPUB extraction
- BookImport and ChapterIngester wired in runner main.go
- CreateImportTask interface gains initiatorUserID param; threaded through store/asynq/handler
- domain.ImportTask gains InitiatorUserID field; parseImportTask populates it
- Runner notifies initiator (falls back to 'admin' when empty)
- UI: import task list polls every 3s while any task is pending/running
- UI: label[for] + input[id] fix for a11y warnings in admin import page
- Import page: add file upload with review step before committing
- Backend: add analyze endpoint to preview chapters before import
- Add notifications collection + API for admin alerts
- Add bell icon in header with notification dropdown (admin only)
- Runner creates notification on import completion
- Notifications link to /admin/import for easy review
- Add ImportTask/ImportResult types to domain.go
- Add TypeImportBook to asynqqueue for task routing
- Add CreateImportTask to producer and storage layers
- Add ClaimNextImportTask/FinishImportTask to Consumer interfaces
- Add import task handling to runner (polling + Asynq handler)
- Add BookImporter interface to bookstore for PDF/EPUB parsing
- Add backend API endpoints: POST/GET /api/admin/import
- Add SvelteKit UI at /admin/import with task list
- Add nav link in admin layout
Note: PDF/EPUB parsing is a placeholder - needs external library integration.
- Admin layout: SVG icons, active highlight, divider between nav sections
- Scrape page: status filter pills with counts, text + status combined search
- Audio page: status filter pills, cancel jobs, retry failed jobs, mobile cards for cache tab
- Translation page: status filter pills (incl. cancelled), cancel + retry jobs, mobile cancel/retry cards, i18n for all labels
- AI Jobs page: fix concurrent cancel (Set instead of single slot), per-job cancel errors inline, full mobile card layout, i18n title/heading
- Text-gen page: tagline editable input + copy, warnings copy, i18n title/heading
- Book page: chapter cover Save button, audio monitor link, currentShelf pre-populated from server
- pocketbase.ts: add getBookShelf(), shelf field on UserLibraryEntry
- New API route: POST /api/admin/translation/bulk (proxy for translation retry)
- i18n: 15 new admin_translation_*, admin_ai_jobs_*, admin_text_gen_* keys across all 5 locales
- New `ai_jobs` PocketBase collection tracks all long-running AI tasks
(batch-covers, chapter-names) with status, progress, and cancellation
- `handlers_aijobs.go`: GET/cancel endpoints for ai_jobs; centralised
cancel registry (moved from handlers_catalogue)
- Batch-covers and chapter-names SSE handlers now create/resume ai_job
records, support from_item/to_item ranges, and resume from items_done
on restart via job_id
- New `POST /api/admin/image-gen/auto-prompt`: generates an image prompt
from book description (cover) or chapter title (chapter) via LLM
- image-gen page: "Auto-prompt" button calls auto-prompt API when a slug
is selected; falls back gracefully if TextGen not configured
- text-gen chapter-names: from/to chapter range inputs + job ID display
- catalogue-tools batch-covers: from/to item range + resume job ID input
- pb-init-v3.sh: adds ai_jobs collection (idempotent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chapters_idx was missing created/updated columns (never defined in the
PocketBase schema), causing PocketBase to return 400 for any query
sorted by -created. recentlyUpdatedBooks() uses this sort.
- Add created date field to chapters_idx schema in pb-init-v3.sh
(also added via add_field for existing installations)
- Add idx_chapters_idx_created index for sort performance
- Set created timestamp on first insert in upsertChapterIdx so new
chapters are immediately sortable; existing records retain empty created
and will sort to the back (acceptable — only affects home page recency)
- Fix upsertChapterIdx race: use conflict-retry pattern (mirrors WriteMetadata)
so concurrent goroutines don't double-POST the same chapter number
- Add DeduplicateChapters to BookWriter interface and Store implementation;
keeps the latest record per (slug, number) and deletes extras
- Wire POST /api/admin/dedup-chapters/{slug} handler in server.go
- storage/pocketbase.go: replace http.DefaultClient (no timeout) with a
dedicated pbHTTPClient{Timeout: 30s} so a slow/hung PocketBase cannot
stall the backend or runner indefinitely
- runner/asynq_runner.go: heartbeat ticker was firing at StaleTaskThreshold
(2 min) == the Docker healthcheck deadline, so a single missed tick would
mark the container unhealthy; halved to StaleTaskThreshold/2 (1 min)
- New /discover page with swipe UI: left=skip, right=like, up=read now, down=nope
- Onboarding modal to collect genre/status preferences (persisted in localStorage)
- 3-card stack with pointer-event drag, CSS fly-out animation, 5 action buttons
- Tap card for preview modal; empty state with deck reset
- Like/read-now auto-saves book to user library
- POST /api/discover/vote + DELETE for deck reset
- Discovery vote persistence via PocketBase discovery_votes collection
- Fix duplicate books: dedup by slug in getBooksBySlugs
- Fix WriteMetadata TOCTOU race: conflict-retry on concurrent insert
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add StreamAudioWAV() to pocket-tts and Kokoro clients; pocket-tts streams
raw WAV directly (no ffmpeg), Kokoro requests response_format:wav with stream:true
- GET /api/audio-stream supports ?format=wav for lower-latency first-byte delivery;
WAV cached separately in MinIO as {slug}/{n}/{voice}.wav
- Add GET /api/admin/audio/jobs with optional ?slug filter
- Add POST /api/admin/audio/bulk {slug, voice, from, to, skip_existing, force}
where skip_existing=true (default) resumes interrupted bulk jobs
- Add POST /api/admin/audio/cancel-bulk {slug} to cancel all pending/running tasks
- Add CancelAudioTasksBySlug to taskqueue.Producer + asynqqueue implementation
- Add AudioObjectKeyExt to bookstore.AudioStore for format-aware MinIO keys
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add GET /api/audio-stream/{slug}/{n}?voice= that streams MP3 audio to the
client as TTS generates it, while simultaneously uploading to MinIO. On
subsequent requests the endpoint redirects to the presigned MinIO URL,
skipping generation entirely.
- PocketTTS: StreamAudioMP3 pipes live WAV response body through ffmpeg
(streaming transcode — no full-buffer wait)
- Kokoro: StreamAudioMP3 uses stream:true mode, returning MP3 frames
directly without the two-step download-link flow
- AudioStore: PutAudioStream added for multipart MinIO upload from reader
- WriteTimeout bumped 60s → 15min to accommodate full-chapter streams
- X-Accel-Buffering: no header disables Caddy/nginx response buffering
- LibreTranslate client (chunks on blank lines, ≤4500 chars, 3-goroutine semaphore)
- Runner translation task loop (OTel, heartbeat, MinIO storage)
- PocketBase translation_jobs collection support (create/claim/finish/list)
- Per-chapter language switcher on chapter reader (EN/RU/ID/PT/FR, polls until done)
- Admin /admin/translation page: bulk enqueue form + live-polling jobs table
- New backend routes: POST /api/translation/{slug}/{n}, GET /api/translation/status,
GET /api/translation/{slug}/{n}, GET /api/admin/translation/jobs,
POST /api/admin/translation/bulk
- ListTranslationTasks added to taskqueue.Reader interface + store impl
- All builds and tests pass; svelte-check: 0 errors
- Add POCKET_TTS_URL env to backend service in docker-compose.yml so
pocket-tts voices appear in the voice selector (Doppler secret existed
but the env var was never passed to the container)
- Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of
single-quoted string, causing the duplicate-task guard to always miss
- Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues
TTS internally on 404; AudioPlayer now skips the redundant POST and
polls directly, eliminating the 500 from the PB unique-key conflict
Two bugs caused audio tasks to loop endlessly:
1. claimRecord never set heartbeat_at — newly claimed tasks had
heartbeat_at=null, which matched the reaper's stale filter
(heartbeat_at=null || heartbeat_at<threshold). Tasks were reaped
and reset to pending within seconds of being claimed, before the
30s heartbeat goroutine had a chance to write a timestamp.
Fix: set heartbeat_at=now() in claimRecord alongside status=running.
2. Audio semaphore was checked AFTER claiming the task. When the
semaphore was full the select/break only broke the inner select,
not the for loop — the code fell through and launched an uncapped
goroutine that blocked forever on <-audioSem drain. The task also
stayed status=running with no heartbeat, feeding bug #1.
Fix: pre-acquire a semaphore slot BEFORE claiming the task; release
it immediately if the queue is empty or claim fails.
- Runner fetches 9 browse combos (genre×sort×status) every 6h and stores
JSON snapshots in MinIO libnovel-browse bucket (browse_refresh.go)
- Backend handleBrowse reads page-1 results from MinIO first; falls back
to live novelfire.net fetch; returns empty+cached:false on total failure
instead of 502
- Add BrowseStore interface (bookstore.go), MinIO put/get helpers (minio.go),
Store methods + compile-time assertion (store.go), BucketBrowse config,
wiring in cmd/backend and cmd/runner, docker-compose-new bucket init
- Fix ReapStaleTasks: PocketBase datetime fields require heartbeat_at=null
(not heartbeat_at="") in filter expressions, and nil (not "") in patch
payload — was causing 400 errors on every reap cycle
- backend/: Go API server and runner binaries with PocketBase + MinIO storage
- ui-v2/: SvelteKit frontend rewrite
- docker-compose-new.yml: compose file for the v2 stack
- .gitea/workflows/release-v2.yaml: CI/CD for backend, runner, and ui-v2 Docker Hub images
- scripts/pb-init.sh: migrate from wget to curl, add superuser bootstrap for fresh installs
- .env.example: document DOCKER_BUILDKIT=1 for Colima users
