- Remove SingleFile CLI and Node.js from Dockerfile; runtime base reverted to alpine:3.21
- Replace triggerBrowseSnapshot with triggerDirectScrape: plain Go HTTP GET to novelfire.net
(page is server-rendered, no browser needed)
- Fix Accept-Encoding bug: stop setting header manually so Go transport auto-decompresses gzip
- Add in-memory browse cache fallback (browseMemCache) for when MinIO and upstream both fail
- Add warmBrowseCache() startup goroutine
- Call triggerDirectScrape on MinIO cache hits too, so PocketBase ranking records are
repopulated after a schema reset or fresh deploy without waiting for a cache miss
- Fix grid view cards: wrap in <a> instead of <div> so clicking navigates to book detail
- Remove SINGLEFILE_PATH and BROWSERLESS_URL env vars from Dockerfile
The previous implementation sent a single request with perPage=500 and
silently dropped any records beyond that. Books with 900+ chapters were
truncated to 500 in the chapter list and reader prev/next navigation.
Now loops through all pages until totalItems is exhausted.
- Speed is no longer part of Kokoro generation, in-memory cache keys, or
MinIO object keys — audio is always generated at 1.0 and playback speed
is applied client-side via audioEl.playbackRate
- presignAudio() now calls rewriteHost() so chapter audio URLs use the
public MinIO endpoint (same as presignVoiceSample already did)
- docker-compose.yml: rename MINIO_PUBLIC_ENDPOINT → PUBLIC_MINIO_PUBLIC_URL
for the ui service so SvelteKit's $env/dynamic/public picks it up
Add warmVoiceSamples() goroutine launched from ListenAndServe.
It waits up to 30s for Kokoro to become reachable, then generates
missing voice sample clips for all available voices and uploads them
to MinIO (_voice-samples/{voice}.mp3). Already-existing samples are
skipped, so the operation is idempotent on restarts.
- Fix speed/voice bug: AudioPlayer no longer accepts speed/voice as props;
startPlayback() reads audioStore.voice/speed directly instead of overwriting them
- Add GET /api/voices endpoint (Go) proxying Kokoro, cached in-memory
- Add POST /api/audio/voice-samples endpoint (Go) to pre-generate sample clips
for all voices and store them in MinIO under _voice-samples/{voice}.mp3
- Add GET /api/presign/voice-sample/{voice} endpoint (Go)
- Add SvelteKit proxy routes: /api/voices, /api/presign/voice-sample, /api/audio/voice-samples
- Add presignVoiceSample() helper in minio.ts with proper host rewrite
- Pass book.cover through +page.server.ts -> +page.svelte -> AudioPlayer
- Set navigator.mediaSession.metadata on playback start so cover art,
book title, and chapter title appear on phone lock screen / notification
Admins see a Rescrape button next to the chapter list header.
Clicking it POSTs to /api/scrape with the book's source_url,
then shows an inline status banner (queued / busy / error).
isAdmin is exposed from the server load; no new routes needed.
The boolean flag was set by onended before goto() resolved, causing the
still-mounted outgoing chapter's AudioPlayer $effect to fire and call
startPlayback() for the wrong (old) chapter — restarting it from scratch.
Replace with autoStartChapter (number | null): the AudioPlayer only acts
when its own chapter prop === autoStartChapter, so the outgoing component
never matches and the incoming one fires exactly once on mount.
The stale-prefetch reset effect compared prefetchedFor against nextChapter
only, so when landing on chapter N+1 (prefetchedFor=N+1, nextChapter=N+2)
it would destroy the pre-fetched URL before startPlayback() could use it,
breaking every auto-next transition after the first.
Fix: also allow prefetchedFor === chapter (current page) so the URL
survives long enough to be consumed, then only wipe truly foreign values.
When autoNext is on, kick off prefetchNext() as soon as the current
chapter begins playing (all three success paths in startPlayback).
This gives the full chapter duration as lead time for Kokoro TTS
instead of only the last 10%, making auto-next transitions seamless.
The existing 90%-mark $effect is kept as a fallback for cases where
autoNext is toggled on mid-playback after startPlayback has returned.
- Fix audio presign 404: MinIO upload is now synchronous before response is sent,
eliminating the race where presign was called before the file landed in MinIO
- Replace all Browserless usage with direct HTTP client across catalogue, metadata,
ranking, and browse — novelfire.net pages are server-rendered and don't need a
headless browser; direct is faster and more reliable
- Harden handleBrowse with 3-attempt retry loop, proper backoff, and full
browser-like headers to reduce 502s from novelfire.net bot detection
- Remove Browserless env vars (BROWSERLESS_URL/TOKEN/STRATEGY) from main.go;
add SCRAPER_TIMEOUT as a single timeout knob
- Clean up now-dead rejectResourceTypes var and Browserless-specific WaitFor/
RejectResourceTypes/GotoOptions fields from scraper calls
- Add NextStatus type and prefetch state (nextStatus, nextAudioUrl, nextProgress, nextChapterPrefetched) to AudioStore
- AudioPlayer triggers prefetchNext() when currentTime/duration >= 0.9, autoNext is on, and a next chapter exists
- startPlayback() uses the pre-fetched URL if available (skipping presign round-trip and generation wait)
- Auto-next button in layout shows a pulsing dot while prefetching and a green dot when ready
- AudioPlayer shows inline prefetch progress and ready state below the controls
- Fix indentation regression in layout onended handler
- ScrapeCatalogue: use li.novel-item (was div), extract href from outer <a> and title from h4.novel-title (was h3), detect next page via rel=next (was class=next)
- handleBrowse/triggerBrowseSnapshot: treat zero-byte MinIO cache entries as misses, skip storing empty SingleFile output
- runAsync: after a successful full-catalogue run, call ScrapeRanking and upsert each result into PocketBase ranking collection
- Clear autoStartPending if goto() fails to avoid spurious auto-starts on future navigations
- Clear audioStore.nextChapter on AudioPlayer unmount so a stale chapter can't trigger navigation after leaving a chapter page
- Remove redundant nextChapter write in startPlayback() — the already keeps it in sync
- Replace double inline player controls with compact 'now playing' indicator when mini-bar is active
- Add user_settings PocketBase collection (auto_next, voice, speed) and audio_time field on progress
- Add getSettings/saveSettings/setAudioTime/getAudioTime server functions
- Add GET/PUT /api/settings and GET/PATCH /api/progress/audio-time API routes
- Load settings server-side in layout and apply to audioStore on mount
- Debounced 800ms effect persists settings changes to DB
- Save audio currentTime on pause/end; restore position when replaying a chapter
When autoNext is enabled, audio automatically advances to the next chapter
when a track ends — navigating the page and starting the new chapter's audio.
- audioStore: add autoNext (toggle flag), nextChapter (written by AudioPlayer),
and autoStartPending (set before goto, cleared after auto-start fires)
- layout: update onended to call goto() and set autoStartPending when autoNext
is on and nextChapter is available; add auto-next toggle button to mini-player
bar (double-chevron icon, amber when active)
- AudioPlayer: accept nextChapter prop; write it to audioStore via $effect;
auto-start playback when autoStartPending is set on component mount; show
inline 'Auto' toggle button in ready controls when a next chapter exists
- Chapter page: pass data.next as nextChapter prop to AudioPlayer
Two bugs caused the start/stop loop:
1. The <audio> element was wrapped in {#if audioStore.audioUrl}, so whenever
any reactive state changed (e.g. currentTime ticking), Svelte could destroy
and recreate the element, firing onpause and then the URL effect restarting
playback.
2. Comparing audioEl.src !== url is unreliable — browsers normalise the src
property to a full absolute URL, causing false mismatches every tick.
Fix: make <audio> always present in the DOM (display:none), and track the
loaded URL in a plain local variable (loadedUrl) instead of reading audioEl.src.
- Add audio.svelte.ts: module singleton AudioStore (Svelte 5 runes) with
slug/chapter/title metadata, status, progress, playback state, and
toggleRequest/seekRequest signals for layout<->audio element communication
- Rewrite AudioPlayer.svelte as a store controller: no <audio> element owned;
drives audioStore for presign->generate->play flow; shows inline controls
when current chapter is active, 'Now playing / Load this chapter' banner
when a different chapter is playing
- Update +layout.svelte: single persistent <audio> outside {#key} block so
it never unmounts on navigation; effects to load URL, sync speed, handle
toggle/seek requests; fixed bottom mini-player bar with seek, skip 15s/30s,
speed cycle, go-to-chapter link, dismiss; pb-24 padding when active
- Pass chapterTitle and bookTitle from chapter page to AudioPlayer
handlePresignAudio now checks AudioExists before presigning; previously it
would generate a valid-looking signed URL for a non-existent object, causing
the browser to get 404 from MinIO directly.
- Add AudioExists to Store interface and HybridStore
- Guard handlePresignAudio with AudioExists check, return 404 if missing
- Propagate 404 through presignAudio() in minio.ts (typed error.status=404)
- SvelteKit presign route forwards 404 to the browser instead of 500
- Single 'Play narration' button: checks presign first, plays immediately if audio exists, otherwise triggers generation
- During generation shows an animated pseudo progress bar: slow start (4%/s) → accelerates (12%/s at 30%) → slows near 80% (4%/s) → crawls to 99% (0.3%/s)
- When generation completes, bar jumps to 100% then transitions to audio player
- Auto-plays audio after generation completes
On every scraper startup, EnsureMigrations fetches each collection schema
and PATCHes in any missing fields. This repairs the progress collection on
cloud deploys where pb-init ran before user_id was added to the schema.
No-ops when the field already exists.
The previous ensure_fields helper used python3 which is not available in
alpine:3.19, causing exit 127 in the cloud init container.
Replaced with ensure_field (per-field, no python/jq) that uses only
busybox sh, wget, sed, and awk. Also fixed the HTTP status grep pattern
in create_collection and ensure_field to match only the response header
line (^ HTTP/) instead of the wget error line, eliminating the spurious
'unexpected status server' warnings.
The progress collection was created before user_id was added to the schema,
causing all user-keyed progress queries to return 400. Add an ensure_fields
helper that idempotently patches existing collections with missing fields,
and run it for progress.user_id on every init.
Replace the single-button placeholder with a full homepage:
- Stats bar: total books, total chapters, books in progress
- Continue Reading grid (up to 6): links directly to last chapter read
- Recently Updated grid (up to 6): recent books not already in progress
- Empty state with Discover Novels CTA when library is empty
New pocketbase.ts helpers: recentlyAddedBooks(), getHomeStats(),
listN(), countCollection().
The pre-compiled single-file binary requires glibc symbols (__res_init)
that Alpine's gcompat shim does not provide, causing exit status 127.
Switch runtime base to node:22-alpine and install single-file-cli via npm.
Also add .dockerignore to exclude bin/ and static/ from build context.
- New MinIO bucket 'libnovel-browse' (MINIO_BUCKET_BROWSE env) for storing
self-contained HTML snapshots of novelfire browse pages
- Store interface gains SaveBrowsePage / GetBrowsePage / BrowsePageKey methods
- handleBrowse is now cache-first: serves from MinIO snapshot when available,
then fires a background triggerBrowseSnapshot goroutine to populate cache
on live-fetch (de-duplicated, 90s timeout)
- New 'save-browse' CLI subcommand to bulk-capture pages via SingleFile CLI
- Dockerfile: downloads pinned single-file-x86_64-linux binary (v2.0.83),
adds gcompat + libstdc++ to Alpine runtime for glibc compatibility
- docker-compose: adds libnovel-browse bucket init and SINGLEFILE_PATH env
- .gitignore: exclude scraper/scraper build artifact
The /ranking route is removed. /browse becomes the Discover page with:
- Sort=Ranking option: fetches from /api/ranking (richer metadata — author,
genres, source_url) and renders the full pre-computed ranked list
- Sort=Popular/New/Updated: fetches from /api/browse as before, paginated
- Grid / List view toggle: grid is the default for browse; list auto-selects
for ranking and gives the information-dense ranked layout (was /ranking)
- Admin Refresh catalogue button (moved from /ranking) and per-novel Scrape
button work in both views
- Genre and status filters are disabled (visually dimmed) when sort=rank
since the ranking endpoint does not support per-page filtering
- Nav: 'Browse' renamed to 'Discover', 'Ranking' link removed
Add optional user_id field to the progress collection. When a user is
authenticated, progress is keyed by user_id instead of session_id, making
it portable across devices and browsers. Anonymous reading still works via
session_id as before.
On login or registration, any progress accumulated under the anonymous
session is merged into the user's account (most-recent timestamp wins),
so chapters read before logging in are not lost.
- scripts/pb-init.sh: add user_id field to progress collection schema
- scraper/internal/storage/pocketbase.go: add user_id to EnsureCollections
- ui/src/lib/server/pocketbase.ts: Progress interface gains user_id;
getProgress/allProgress/setProgress accept optional userId and query/write
by user_id when present; add mergeSessionProgress() helper
- ui/src/routes/login/+page.server.ts: call mergeSessionProgress after
successful login and registration (fire-and-forget, non-fatal)
- ui/src/routes/api/progress/+server.ts: pass locals.user?.id to setProgress
- ui/src/routes/books/+page.server.ts: pass locals.user?.id to allProgress
- ui/src/routes/books/[slug]/+page.server.ts: pass locals.user?.id to getProgress
Add a second MinIO client (pub) initialized with MINIO_PUBLIC_ENDPOINT so
presigned audio URLs are signed against the public hostname from the start,
rather than signed internally and then rewritten. This avoids AWS4 signature
mismatch (SignatureDoesNotMatch 403) that occurred when the signed host was
substituted after signing.
- storage/minio.go: add PublicEndpoint/PublicUseSSL to MinioConfig; add pub
client field; NewMinioClient creates pub client when public endpoint differs;
PresignAudio uses pub, PresignChapter keeps internal client
- cmd/scraper/main.go: wire MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL env vars
- docker-compose.yml: expose MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL to scraper service
- ui/src/lib/server/minio.ts: remove rewriteHost() call from presignAudio
The SvelteKit server fetches chapter markdown server-side using the presigned
URL. rewriteHost() was replacing the internal minio:9000 host with the public
PUBLIC_MINIO_PUBLIC_URL (localhost:9000), which is unreachable from inside
Docker, causing MinIO 403/connection errors.
presignChapter now takes an optional rewrite=false parameter — the server-side
load function gets the raw internal URL, while any future browser-facing use
can pass rewrite=true to get the public-facing URL.
- ui/Dockerfile: copy package-lock.json and run npm ci --omit=dev in the
runtime stage so marked (and other runtime deps) are available to
adapter-node at startup — fixes ERR_MODULE_NOT_FOUND for 'marked'
- storage: add ReindexChapters to Store interface and HybridStore — walks
MinIO objects for a slug, reads chapter titles, upserts chapters_idx
- server: add POST /api/reindex/{slug} to rebuild chapters_idx from MinIO
Replace selected={data.x === opt.value} on individual <option> elements
with value={data.x} on the <select> — the idiomatic Svelte approach that
ensures correct hydration and form submission of the current filter values.
Adds /ranking route showing the cached ranking list with cover, title,
author, status, and genres. Admin users get a Refresh button that
triggers a full catalogue scrape. Adds Ranking to the nav.
runAsync creates a scraping_tasks record on job start, flushes progress
counters via OnProgress, and finalizes status (done/failed/cancelled) on
completion. Adds GET /api/scrape/tasks to list all historical jobs.
Also fixes relative cover URLs in parseBrowsePage.
Adds atomic counters for books_found, chapters_scraped, chapters_skipped,
and errors. The new OnProgress callback fires after each counter update
and once more at the end of Run, giving callers a live progress feed.
Adds a scraping_tasks PocketBase collection with fields for kind, status,
progress counters, timestamps, and error info. Exposes CreateScrapeTask,
UpdateScrapeTask, and ListScrapeTasks on the Store interface with
implementations in HybridStore and PocketBaseStore.
- Inject *slog.Logger into HybridStore, PocketBaseStore, and pbClient
- Fix credential defaults in main.go (changeme123 / admin) to match docker-compose
- listOne/listAll/upsert/deleteWhere now return errors on non-2xx HTTP status
- WriteChapter: log warn instead of discarding UpsertChapterIdx error
- MetadataMtime, GetAudioCache, GetProgress: log warn on PocketBase failures
- EnsureCollections: log info/debug/warn per outcome instead of _ = err
- CountChapterIdx: log warn on failure instead of silently returning 0
- server: log warn when SetAudioCache fails after audio generation
- NewHybridStore: add explicit Ping() before EnsureCollections for fast-fail on bad credentials
novelfire.net chapter-list pages (/chapters?page=N) are server-rendered —
verified via curl. Switch urlClient to NewDirectHTTPClient alongside the
existing chapterClient. Remove BROWSERLESS_URL_STRATEGY env var and clean
up the now-irrelevant WaitFor/GotoOptions fields from both ScrapeChapterList
and ScrapeChapterListPage ContentRequests.
novelfire.net chapter content is server-rendered, so Browserless is not
needed. Add a dedicated chapterClient (always StrategyDirect) to Scraper
and use it in ScrapeChapterText, removing the now-irrelevant WaitFor /
RejectResourceTypes / GotoOptions fields from the ContentRequest.
