Two bugs fixed in runScrapeTask / runCatalogueTask:
1. FinishScrapeTask was called with the task's own context, which is
already cancelled when the task is stopped. The PATCH to PocketBase
failed silently, leaving all counters at their initial zero values.
Fix: use a fresh context.WithTimeout(Background, 15s) for the write.
2. BooksFound was double-counted: RunBook already sets BooksFound=1 on
success, but the accumulation loop added an extra +1 unconditionally,
reporting 2 books per successful scrape.
Fix: result.BooksFound += bookResult.BooksFound (drop the + 1).
- Add `archived` bool to domain.BookMeta, pbBook, and Meilisearch bookDoc
- ArchiveBook / UnarchiveBook patch the PocketBase record; ListBooks filters
archived=false so hidden books disappear from all public responses
- Meilisearch: add `archived` as a filterable attribute; Search and Catalogue
always prepend `archived = false` to exclude archived books from results
- DeleteBook permanently removes the PocketBase record, all chapters_idx rows,
MinIO chapter objects, cover image, and the Meilisearch document
- New BookAdminStore interface with ArchiveBook, UnarchiveBook, DeleteBook
- Admin HTTP endpoints: PATCH /api/admin/books/{slug}/archive|unarchive,
DELETE /api/admin/books/{slug}
- PocketBase schema: archived field added to live pb.libnovel.cc and to
pb-init-v3.sh (both create block and add_field migration)
Split chapter text into ~1000-char sentence-boundary chunks before sending
to kokoro-fastapi. Each chunk is generated individually and the raw MP3 bytes
are concatenated. This prevents the EOF / timeout failures that occur when
the server receives a very large single request (e.g. a full PDF 'Full Text'
chapter). chunkText() breaks at sentence endings (. ! ? newlines) to preserve
natural speech flow.
- parsePDF: return all text as single 'Full Text' chapter (admin splits manually)
- parseEPUB: fix chapter numbering to use sequential counter not spine index
- Remove dead code: chaptersFromBookmarks, cleanChapterText, extractChaptersFromText, chapterHeadingRE; drop pdfcpu alias and regexp imports
- Backend: POST /api/admin/books/:slug/split-chapters endpoint — splits text on '---' dividers, optional '## Title' headers, writes chapters via WriteChapter
- UI: admin panel now shows for all admin users regardless of source_url; chapter split tool shown when book has single 'Full Text' chapter, pre-fills from MinIO content
Three fixes to PDF chapter extraction quality:
1. Page ordering: parse page number from pdfcpu filename (out_Content_page_N.txt)
instead of using lexicographic sort index — fixes chapters bleeding into each
other (e.g. Prologue text appearing inside Chapter 1).
2. Windows-1252 chars: map bytes 0x91-0x9F to proper Unicode (curly quotes U+2018/
U+2019/U+201C/U+201D, em-dash U+2014, etc.) instead of raw Latin-1 control
bytes that rendered as ◆ in the browser.
3. Chapter header cleanup: skip the first page of each bookmark range (decorative
title art page) and strip any run-on title fragment at the start of the first
body page (e.g. 'for New Journeys!I stood atop...' → 'I stood atop...'). The
remaining sentence truncation is a fundamental limitation of this PDF's
PUA-encoded body font (C2_1/Literata) — those glyphs cannot be decoded without
the publisher's private ToUnicode mapping.
Use PDF outline (bookmarks) for chapter titles and page ranges, then
extract text from per-page content streams via pdfcpu ExtractContent.
This avoids the indefinite hang caused by dslipak/pdf trying to resolve
custom font ToUnicode CMaps on publisher PDFs.
- parsePDF: decrypt → ExtractContent → Bookmarks → chaptersFromBookmarks
- chaptersFromBookmarks: flatten bookmark tree, skip front/back matter
(Cover, Insert, Title Page, Copyright, Appendix), assign page ranges
- extractTextFromContentStream: handle TJ arrays (concat literal strings,
skip hex glyph arrays and kerning numbers) + single Tj strings
- Falls back to paragraph-splitting when no bookmarks present
- Build verified; test PDF produces 9 chapters with proper titles
- parsePDF function restored in import.go (body was orphaned outside function)
- ParseImportFile() called at upload time with 3-min timeout; chapters stored as JSON in MinIO
- runner.go: prefer ChaptersKey path (read pre-parsed JSON) over BookImport.Import()
- ImportChapterStore interface added; store wired in runner/main.go
- HeartbeatTask and ReapStaleTasks now include import_tasks collection
- parseImportTask now returns ChaptersKey in domain.ImportTask
- asynq_runner.go handleImportTask passes ChaptersKey
- pb-init-v3.sh: chapters_key field added to import_tasks schema
- parsePDF now attempts to strip encryption via pdfcpu (empty user password)
before handing bytes to dslipak/pdf — fixes '256-bit encryption key' error
on publisher PDFs that use owner-only encryption (copy/print restrictions)
- Add pdfcpu v0.11.1 as direct dependency (was already indirect)
- docker-compose.yml minio-init: add 'imports' and 'translations' buckets
so a fresh deploy creates all required buckets
- Add ImportFileStore interface to bookstore package
- Add ImportFileStore field to backend.Dependencies
- Wire ImportFileStore: store in cmd/backend/main.go
- handlers_import.go: use s.deps.ImportFileStore.PutImportFile instead of
broken s.deps.Producer.(*storage.Store) type assertion (fails when Asynq active)
- pb-init-v3.sh: add import_tasks and notifications collection definitions
- Import task: persist object_key, author, cover_url, genres, summary,
book_status in PocketBase so the runner can fetch the file and write
book metadata on completion
- Runner poll mode: pass task.ObjectKey instead of empty string
- Runner: write BookMeta + UpsertBook in Meilisearch after chapter ingest
so imported books appear in catalogue and search
- Import UI: add author, cover URL, genres, summary, status fields; add
AI tasks panel (chapter names, description, image gen, tagline) after
import completes; add AI tasks button on each done task in the list
- Admin nav: add Notifications entry to sidebar (all 5 locales)
- Logout: delete user_sessions row on sign-out so sessions don't
accumulate as phantoms after each login/logout cycle
- docker-compose.yml: BODY_SIZE_LIMIT=52428800 (50MB) on UI service
— adapter-node was rejecting PDFs >512KB with 'Content-length exceeds limit'
- storage/import.go: add AnalyzeFile() public function using real PDF/EPUB parsers
- handlers_import.go: analyzeImportFile() now calls storage.AnalyzeFile() instead
of the file-size stub, so chapter count is accurate in the preview
- NewBookImporter now takes *Store instead of raw *minio.Client (same package access)
- Runner Dependencies gains ChapterIngester interface; storeImportedChapters no-op removed
- store.IngestChapters is now actually called after PDF/EPUB extraction
- BookImport and ChapterIngester wired in runner main.go
- CreateImportTask interface gains initiatorUserID param; threaded through store/asynq/handler
- domain.ImportTask gains InitiatorUserID field; parseImportTask populates it
- Runner notifies initiator (falls back to 'admin' when empty)
- UI: import task list polls every 3s while any task is pending/running
- UI: label[for] + input[id] fix for a11y warnings in admin import page
- Import page: add file upload with review step before committing
- Backend: add analyze endpoint to preview chapters before import
- Add notifications collection + API for admin alerts
- Add bell icon in header with notification dropdown (admin only)
- Runner creates notification on import completion
- Notifications link to /admin/import for easy review
- Add ImportTask/ImportResult types to domain.go
- Add TypeImportBook to asynqqueue for task routing
- Add CreateImportTask to producer and storage layers
- Add ClaimNextImportTask/FinishImportTask to Consumer interfaces
- Add import task handling to runner (polling + Asynq handler)
- Add BookImporter interface to bookstore for PDF/EPUB parsing
- Add backend API endpoints: POST/GET /api/admin/import
- Add SvelteKit UI at /admin/import with task list
- Add nav link in admin layout
Note: PDF/EPUB parsing is a placeholder - needs external library integration.
Cloudflare Workers AI changed the API for flux-2-dev, flux-2-klein-4b,
and flux-2-klein-9b to require multipart/form-data (instead of JSON) and
now returns {"image":"<base64>"} instead of raw PNG bytes.
- Add requiresMultipart() helper for the three FLUX.2 models
- callImageAPI builds multipart body for those models, JSON for others
- Parse {"image":"<base64>"} JSON response; fall back to raw bytes for legacy models
- Use "steps" field name (not "num_steps") in multipart forms per CF docs
- Book page: capture and display actual backend error message instead of blank 'Error'
- Admin layout: SVG icons, active highlight, divider between nav sections
- Scrape page: status filter pills with counts, text + status combined search
- Audio page: status filter pills, cancel jobs, retry failed jobs, mobile cards for cache tab
- Translation page: status filter pills (incl. cancelled), cancel + retry jobs, mobile cancel/retry cards, i18n for all labels
- AI Jobs page: fix concurrent cancel (Set instead of single slot), per-job cancel errors inline, full mobile card layout, i18n title/heading
- Text-gen page: tagline editable input + copy, warnings copy, i18n title/heading
- Book page: chapter cover Save button, audio monitor link, currentShelf pre-populated from server
- pocketbase.ts: add getBookShelf(), shelf field on UserLibraryEntry
- New API route: POST /api/admin/translation/bulk (proxy for translation retry)
- i18n: 15 new admin_translation_*, admin_ai_jobs_*, admin_text_gen_* keys across all 5 locales
speechSynthesis is silently muted on iOS Safari and Chrome Android after
the audio session ends (onended), so chapter announcements never played.
Fix:
- Add GET /api/tts-announce backend endpoint: streams a short TTS clip
for arbitrary text without MinIO caching (backend/internal/backend/)
- Add GET /api/announce SvelteKit proxy route (no paywall)
- Add announceNavigatePending/announcePendingSlug/announcePendingChapter
to AudioStore
- Rewrite onended announce branch: sets audioStore.audioUrl to the
announcement clip URL so the persistent <audio> element plays it;
the next onended detects announceNavigatePending and navigates
- 10s safety timeout in case the clip fails to load/end
- Add POST /api/admin/image-gen/async: fire-and-forget image generation
that stores the result (base64) in an ai_job payload and returns 202
immediately — no more 60-120s blocking on FLUX models
- Add POST /api/admin/text-gen/description/async: same pattern for book
description generation
- Register both new routes in server.go
- Rewrite image-gen admin page to use the async path (submit → redirect
to AI Jobs for monitoring)
- Extend ai-jobs page with Review panels for image-gen jobs (show image,
Save as cover / Download / Discard) and description jobs (diff old vs
new, editable textarea, Apply / Discard)
- Add POST /api/admin/text-gen/chapter-names/async backend endpoint: fire-and-forget,
returns job_id immediately (HTTP 202), runs batch generation in background goroutine,
persists proposed titles in ai_job payload when done
- Register new route in server.go alongside existing SSE endpoint (backward compat)
- Add SvelteKit proxy at /api/admin/text-gen/chapter-names/async
- Add SvelteKit proxy for GET /api/admin/ai-jobs/[id] (job detail with payload)
- Add Review button on ai-jobs page for done chapter-names jobs; inline panel shows
editable title table (old to new) with Apply All button that POSTs to chapter-names/apply
- Backend: add GET /api/audio-preview/{slug}/{n} — generates first ~1800-char
chunk via CF AI so playback starts immediately; full chapter cached in MinIO
- Frontend: replace CF AI spinner with preview blob URL + background swap to
full presigned URL when runner finishes, preserving currentTime
- AudioPlayer: isPreview state + 'preview' badge in mini-bar during swap
- pocketbase.ts: fix 403 on stale token — reduce TTL to 50 min + retry once
on 401/403 with forced re-auth (was cached 12 h, PB tokens expire in 1 h)
- Footer build time now rendered in user's local timezone via toLocaleString()
- New `ai_jobs` PocketBase collection tracks all long-running AI tasks
(batch-covers, chapter-names) with status, progress, and cancellation
- `handlers_aijobs.go`: GET/cancel endpoints for ai_jobs; centralised
cancel registry (moved from handlers_catalogue)
- Batch-covers and chapter-names SSE handlers now create/resume ai_job
records, support from_item/to_item ranges, and resume from items_done
on restart via job_id
- New `POST /api/admin/image-gen/auto-prompt`: generates an image prompt
from book description (cover) or chapter title (chapter) via LLM
- image-gen page: "Auto-prompt" button calls auto-prompt API when a slug
is selected; falls back gracefully if TextGen not configured
- text-gen chapter-names: from/to chapter range inputs + job ID display
- catalogue-tools batch-covers: from/to item range + resume job ID input
- pb-init-v3.sh: adds ai_jobs collection (idempotent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CF Workers AI has a ~4MB JSON body limit. Large cover images base64-encoded
can easily exceed this, causing Cloudflare to return a 502 Bad Gateway.
Added resizeRefImage() which scales the reference down so its longest side
≤ 768px (nearest-neighbour, re-encoded as JPEG) before passing it to the
GenerateImageFromReference call. Images already within the limit pass through
unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Llama 4 Scout returns `result.response` as an array of objects
[{"generated_text":"..."}] instead of a plain string. Decode into
json.RawMessage and try both shapes; fall back to generated_text[0].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split chapter-name LLM requests into 100-chapter batches and stream
results back as SSE so large books (e.g. Shadow Slave: 2916 chapters)
never time out or truncate. Frontend shows live batch progress inline
and accumulates proposals as they arrive.
chapters_idx was missing created/updated columns (never defined in the
PocketBase schema), causing PocketBase to return 400 for any query
sorted by -created. recentlyUpdatedBooks() uses this sort.
- Add created date field to chapters_idx schema in pb-init-v3.sh
(also added via add_field for existing installations)
- Add idx_chapters_idx_created index for sort performance
- Set created timestamp on first insert in upsertChapterIdx so new
chapters are immediately sortable; existing records retain empty created
and will sort to the back (acceptable — only affects home page recency)
- Fix upsertChapterIdx race: use conflict-retry pattern (mirrors WriteMetadata)
so concurrent goroutines don't double-POST the same chapter number
- Add DeduplicateChapters to BookWriter interface and Store implementation;
keeps the latest record per (slug, number) and deletes extras
- Wire POST /api/admin/dedup-chapters/{slug} handler in server.go
The aura-2-en model enforces a hard 2 000-character limit per request.
Chapters routinely exceed this, producing 413 errors.
GenerateAudio now splits the stripped text into ≤1 800-char chunks at
paragraph → sentence → space → hard-cut boundaries, calls the API once
per chunk, and concatenates the MP3 frames. Callers (runner, streaming
handler) are unchanged. StreamAudioMP3/WAV inherit the fix automatically
since they delegate to GenerateAudio.
- Default max_tokens to 4096 for chapter-names so large chapter lists
are not cut off mid-JSON by the model's token limit
- Rewrite system prompt to clarify placeholder semantics ({n} = number,
{scene} = scene hint) and explicitly forbid echoing the number inside
the title field — prevents "Chapter 1 - 1: ..." style duplications
- UI: surface raw_response in the error area when chapters:[] is returned
so the admin can see what the model actually produced
Adds backend handlers and SvelteKit UI for an admin text generation tool.
The tool lets admins propose and apply AI-generated chapter titles and book
descriptions using Cloudflare Workers AI (12 LLM models, model selector shared
across both tabs).
**Ratings (1–5 stars)**
- New `book_ratings` PB collection (session_id, user_id, slug, rating)
- `getBookRating`, `getBookAvgRating`, `setBookRating` in pocketbase.ts
- GET/POST /api/ratings/[slug] API route
- StarRating.svelte component with hover, animated stars, avg display
- Star rating shown on book detail page (desktop + mobile)
**Plan-to-Read shelf**
- `shelf` field added to `user_library` (reading/plan_to_read/completed/dropped)
- `updateBookShelf`, `getShelfMap` in pocketbase.ts
- PATCH /api/library/[slug] for shelf updates
- Shelf selector dropdown on book detail page (only when saved)
- Shelf tabs on library page to filter by category
**Sleep timer**
- `sleepUntil` state added to AudioStore
- Layout handles timer lifecycle (survives chapter navigation)
- Cycles Off → 15m → 30m → 45m → 60m → Off
- Shows live countdown in AudioPlayer when active
**EPUB export**
- Go backend: GET /api/export/{slug}?from=N&to=N
- Generates valid EPUB2 zip (mimetype uncompressed, OPF, NCX, XHTML chapters)
- Markdown → HTML via goldmark
- SvelteKit proxy at /api/export/[slug]
- Download button on book detail page (only when in library)
**Fix TS errors**
- discover/+page.svelte: currentBook possibly undefined (use {@const book})
- cardEl now $state for reactive binding
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ClaimNextTranslationTask and HeartbeatTask were no-ops in the asynq
Consumer, so translation tasks created in PocketBase were never picked
up by the runner. Translation tasks live in PocketBase (not Redis),
so they must be claimed/heartbeated via the underlying pb consumer.
ReapStaleTasks is also delegated so stale translation tasks get reset.
Also removes the LibreTranslate healthcheck from homelab/runner
docker-compose.yml and relaxes depends_on to service_started — the
healthcheck was blocking runner startup until models loaded (~2 min)
and the models are already pre-downloaded in the volume.
- storage/pocketbase.go: replace http.DefaultClient (no timeout) with a
dedicated pbHTTPClient{Timeout: 30s} so a slow/hung PocketBase cannot
stall the backend or runner indefinitely
- runner/asynq_runner.go: heartbeat ticker was firing at StaleTaskThreshold
(2 min) == the Docker healthcheck deadline, so a single missed tick would
mark the container unhealthy; halved to StaleTaskThreshold/2 (1 min)
- New /discover page with swipe UI: left=skip, right=like, up=read now, down=nope
- Onboarding modal to collect genre/status preferences (persisted in localStorage)
- 3-card stack with pointer-event drag, CSS fly-out animation, 5 action buttons
- Tap card for preview modal; empty state with deck reset
- Like/read-now auto-saves book to user library
- POST /api/discover/vote + DELETE for deck reset
- Discovery vote persistence via PocketBase discovery_votes collection
- Fix duplicate books: dedup by slug in getBooksBySlugs
- Fix WriteMetadata TOCTOU race: conflict-retry on concurrent insert
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add StreamAudioWAV() to pocket-tts and Kokoro clients; pocket-tts streams
raw WAV directly (no ffmpeg), Kokoro requests response_format:wav with stream:true
- GET /api/audio-stream supports ?format=wav for lower-latency first-byte delivery;
WAV cached separately in MinIO as {slug}/{n}/{voice}.wav
- Add GET /api/admin/audio/jobs with optional ?slug filter
- Add POST /api/admin/audio/bulk {slug, voice, from, to, skip_existing, force}
where skip_existing=true (default) resumes interrupted bulk jobs
- Add POST /api/admin/audio/cancel-bulk {slug} to cancel all pending/running tasks
- Add CancelAudioTasksBySlug to taskqueue.Producer + asynqqueue implementation
- Add AudioObjectKeyExt to bookstore.AudioStore for format-aware MinIO keys
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The admin_nav_* message files generated by paraglide used 0-param inner
functions but called them with inputs, causing 50 svelte-check type errors.
Add _inputs = {} to each inner locale function to match the call signature.
Also adds backend/backend and backend/runner to .gitignore — binaries are
built inside Dockerfile and should never be committed.
- homelab/docker-compose.yml: add redis:7-alpine service (port 6379 bound to host
so Caddy TLS proxy on prod can reach it), add libretranslate service, add
redis_data and libretranslate_data volumes
- asynqqueue/producer.go: Asynq enqueue failures are now logged as warnings instead
of returned as errors — PB record already exists so runner picks it up via poll
- backend/main.go: pass logger to NewProducer
Root cause: Redis was not reachable at 192.168.0.109:6379 because the redis
container had no host port binding. Caddy TLS proxy terminates TLS but could
not TCP-connect to the backend Redis.
Add GET /api/audio-stream/{slug}/{n}?voice= that streams MP3 audio to the
client as TTS generates it, while simultaneously uploading to MinIO. On
subsequent requests the endpoint redirects to the presigned MinIO URL,
skipping generation entirely.
- PocketTTS: StreamAudioMP3 pipes live WAV response body through ffmpeg
(streaming transcode — no full-buffer wait)
- Kokoro: StreamAudioMP3 uses stream:true mode, returning MP3 frames
directly without the two-step download-link flow
- AudioStore: PutAudioStream added for multipart MinIO upload from reader
- WriteTimeout bumped 60s → 15min to accommodate full-chapter streams
- X-Accel-Buffering: no header disables Caddy/nginx response buffering
Two bugs prevented asynq mode from working correctly on the homelab runner:
1. No healthcheck file: asynq mode never writes /tmp/runner.alive, so
Docker healthcheck always fails. Added heartbeat goroutine that
writes the file every StaleTaskThreshold (30s).
2. Translation tasks not dispatched: translation uses ClaimNextTranslationTask
(PocketBase poll queue), not Redis/asynq. Audio + scrape use asynq mux,
but translation sits in PocketBase forever. Added pollTranslationTasks()
goroutine that polls PocketBase on the same PollInterval as the old
poll() loop.
All Go tests pass (go test ./... in backend/).
- LibreTranslate client (chunks on blank lines, ≤4500 chars, 3-goroutine semaphore)
- Runner translation task loop (OTel, heartbeat, MinIO storage)
- PocketBase translation_jobs collection support (create/claim/finish/list)
- Per-chapter language switcher on chapter reader (EN/RU/ID/PT/FR, polls until done)
- Admin /admin/translation page: bulk enqueue form + live-polling jobs table
- New backend routes: POST /api/translation/{slug}/{n}, GET /api/translation/status,
GET /api/translation/{slug}/{n}, GET /api/admin/translation/jobs,
POST /api/admin/translation/bulk
- ListTranslationTasks added to taskqueue.Reader interface + store impl
- All builds and tests pass; svelte-check: 0 errors
novelfire.net changed its book page structure. Old selectors produced empty
status and null genres for every book, causing all Meilisearch filters to
return zero results.
Old → new:
- status: <span class="status"> → <strong class="ongoing|completed|hiatus">
(text lowercased for consistent index values)
- genres: <div class="genres"> <a> → <div class="categories"> <a class="property-item">
(text lowercased for consistent index values)
Adds TestParseMetadataSelectors to guard against future regressions.
handlePresignVoiceSample generates voice samples on demand via pocket-tts,
which requires WAV→MP3 transcoding via ffmpeg. The backend was using
distroless/static (no ffmpeg) so all pocket-tts preview requests returned 500.
Switch backend stage to Alpine + ffmpeg, matching the runner image.
- scraper.go: ScrapeCatalogue now uses retryGet (9 attempts, 10s base) +
500–1500ms inter-page jitter instead of bare GetContent. ScrapeMetadata
also switched to retryGet so a single 429 on a book page is retried rather
than aborting the whole refresh.
- catalogue_refresh.go: per-book delay is now configurable
(RUNNER_CATALOGUE_REQUEST_DELAY, default 2s) + up to 50% random jitter
applied before every metadata fetch. Only metadata is scraped here —
chapters are fetched on-demand, not during catalogue refresh. Progress
logged every 50 books instead of 100.
- config.go / runner.go / main.go: add CatalogueRequestDelay field wired
from RUNNER_CATALOGUE_REQUEST_DELAY env var.
- release.yaml: comment out upload-sourcemaps job and remove it from the
release needs; GlitchTip auth token needs refreshing after DB wipe.
- Add POCKET_TTS_URL env to backend service in docker-compose.yml so
pocket-tts voices appear in the voice selector (Doppler secret existed
but the env var was never passed to the container)
- Fix GetAudioTask PocketBase filter using %q (double-quotes) instead of
single-quoted string, causing the duplicate-task guard to always miss
- Fix AudioPlayer double-POST: GET /api/presign/audio already enqueues
TTS internally on 404; AudioPlayer now skips the redundant POST and
polls directly, eliminating the 500 from the PB unique-key conflict
