- Go scraper: Version/Commit vars in main.go, injected via -ldflags; Server struct + New() updated; GET /health and new GET /api/version expose them
- UI Dockerfile: ARG BUILD_VERSION/BUILD_COMMIT → ENV PUBLIC_BUILD_VERSION/PUBLIC_BUILD_COMMIT for SvelteKit
- Footer: shows version+short commit when not 'dev' (text-zinc-800, subtle)
- docker-compose: args blocks for scraper and ui build sections pass $GIT_TAG/$GIT_COMMIT
- Go scraper: add PresignAvatarUploadURL/PresignAvatarURL/DeleteAvatar to
Store interface, implement on HybridStore+MinioClient, register
GET /api/presign/avatar-upload/{userId} and /api/presign/avatar/{userId}
- SvelteKit: replace direct AWS S3 SDK in minio.ts with presign calls to
the Go scraper; rewrite avatar +server.ts (POST=presign, PATCH=record key)
- iOS: rewrite uploadAvatar() as 3-step presigned PUT flow; refactor
chip components into shared ChipButton in CommonViews.swift
Replace blocking POST /api/audio with a non-blocking 202 flow: the Go
handler immediately enqueues a job in a new `audio_jobs` PocketBase
collection and returns {job_id, status}. A background goroutine runs
the actual Kokoro TTS work and updates job status (pending → generating
→ done/failed). A new GET /api/audio/status/{slug}/{n} endpoint lets
clients poll progress. The SvelteKit proxy and AudioPlayer.svelte are
updated to POST, then poll the status route every 2s until done.
Adds GET /api/search Go endpoint that queries PocketBase books by title/author
substring and fetches novelfire.net /search results via parseBrowsePage(),
merging results with local-first de-duplication. The browse page gains a search
input that navigates to ?q=; results show local vs remote counts and hide
pagination/filter controls while in search mode.
- Remove SingleFile CLI and Node.js from Dockerfile; runtime base reverted to alpine:3.21
- Replace triggerBrowseSnapshot with triggerDirectScrape: plain Go HTTP GET to novelfire.net
(page is server-rendered, no browser needed)
- Fix Accept-Encoding bug: stop setting header manually so Go transport auto-decompresses gzip
- Add in-memory browse cache fallback (browseMemCache) for when MinIO and upstream both fail
- Add warmBrowseCache() startup goroutine
- Call triggerDirectScrape on MinIO cache hits too, so PocketBase ranking records are
repopulated after a schema reset or fresh deploy without waiting for a cache miss
- Fix grid view cards: wrap in <a> instead of <div> so clicking navigates to book detail
- Remove SINGLEFILE_PATH and BROWSERLESS_URL env vars from Dockerfile
- Speed is no longer part of Kokoro generation, in-memory cache keys, or
MinIO object keys — audio is always generated at 1.0 and playback speed
is applied client-side via audioEl.playbackRate
- presignAudio() now calls rewriteHost() so chapter audio URLs use the
public MinIO endpoint (same as presignVoiceSample already did)
- docker-compose.yml: rename MINIO_PUBLIC_ENDPOINT → PUBLIC_MINIO_PUBLIC_URL
for the ui service so SvelteKit's $env/dynamic/public picks it up
Add warmVoiceSamples() goroutine launched from ListenAndServe.
It waits up to 30s for Kokoro to become reachable, then generates
missing voice sample clips for all available voices and uploads them
to MinIO (_voice-samples/{voice}.mp3). Already-existing samples are
skipped, so the operation is idempotent on restarts.
- Fix speed/voice bug: AudioPlayer no longer accepts speed/voice as props;
startPlayback() reads audioStore.voice/speed directly instead of overwriting them
- Add GET /api/voices endpoint (Go) proxying Kokoro, cached in-memory
- Add POST /api/audio/voice-samples endpoint (Go) to pre-generate sample clips
for all voices and store them in MinIO under _voice-samples/{voice}.mp3
- Add GET /api/presign/voice-sample/{voice} endpoint (Go)
- Add SvelteKit proxy routes: /api/voices, /api/presign/voice-sample, /api/audio/voice-samples
- Add presignVoiceSample() helper in minio.ts with proper host rewrite
- Pass book.cover through +page.server.ts -> +page.svelte -> AudioPlayer
- Set navigator.mediaSession.metadata on playback start so cover art,
book title, and chapter title appear on phone lock screen / notification
- Fix audio presign 404: MinIO upload is now synchronous before response is sent,
eliminating the race where presign was called before the file landed in MinIO
- Replace all Browserless usage with direct HTTP client across catalogue, metadata,
ranking, and browse — novelfire.net pages are server-rendered and don't need a
headless browser; direct is faster and more reliable
- Harden handleBrowse with 3-attempt retry loop, proper backoff, and full
browser-like headers to reduce 502s from novelfire.net bot detection
- Remove Browserless env vars (BROWSERLESS_URL/TOKEN/STRATEGY) from main.go;
add SCRAPER_TIMEOUT as a single timeout knob
- Clean up now-dead rejectResourceTypes var and Browserless-specific WaitFor/
RejectResourceTypes/GotoOptions fields from scraper calls
- ScrapeCatalogue: use li.novel-item (was div), extract href from outer <a> and title from h4.novel-title (was h3), detect next page via rel=next (was class=next)
- handleBrowse/triggerBrowseSnapshot: treat zero-byte MinIO cache entries as misses, skip storing empty SingleFile output
- runAsync: after a successful full-catalogue run, call ScrapeRanking and upsert each result into PocketBase ranking collection
handlePresignAudio now checks AudioExists before presigning; previously it
would generate a valid-looking signed URL for a non-existent object, causing
the browser to get 404 from MinIO directly.
- Add AudioExists to Store interface and HybridStore
- Guard handlePresignAudio with AudioExists check, return 404 if missing
- Propagate 404 through presignAudio() in minio.ts (typed error.status=404)
- SvelteKit presign route forwards 404 to the browser instead of 500
- New MinIO bucket 'libnovel-browse' (MINIO_BUCKET_BROWSE env) for storing
self-contained HTML snapshots of novelfire browse pages
- Store interface gains SaveBrowsePage / GetBrowsePage / BrowsePageKey methods
- handleBrowse is now cache-first: serves from MinIO snapshot when available,
then fires a background triggerBrowseSnapshot goroutine to populate cache
on live-fetch (de-duplicated, 90s timeout)
- New 'save-browse' CLI subcommand to bulk-capture pages via SingleFile CLI
- Dockerfile: downloads pinned single-file-x86_64-linux binary (v2.0.83),
adds gcompat + libstdc++ to Alpine runtime for glibc compatibility
- docker-compose: adds libnovel-browse bucket init and SINGLEFILE_PATH env
- .gitignore: exclude scraper/scraper build artifact
- ui/Dockerfile: copy package-lock.json and run npm ci --omit=dev in the
runtime stage so marked (and other runtime deps) are available to
adapter-node at startup — fixes ERR_MODULE_NOT_FOUND for 'marked'
- storage: add ReindexChapters to Store interface and HybridStore — walks
MinIO objects for a slug, reads chapter titles, upserts chapters_idx
- server: add POST /api/reindex/{slug} to rebuild chapters_idx from MinIO
runAsync creates a scraping_tasks record on job start, flushes progress
counters via OnProgress, and finalizes status (done/failed/cancelled) on
completion. Adds GET /api/scrape/tasks to list all historical jobs.
Also fixes relative cover URLs in parseBrowsePage.
- Inject *slog.Logger into HybridStore, PocketBaseStore, and pbClient
- Fix credential defaults in main.go (changeme123 / admin) to match docker-compose
- listOne/listAll/upsert/deleteWhere now return errors on non-2xx HTTP status
- WriteChapter: log warn instead of discarding UpsertChapterIdx error
- MetadataMtime, GetAudioCache, GetProgress: log warn on PocketBase failures
- EnsureCollections: log info/debug/warn per outcome instead of _ = err
- CountChapterIdx: log warn on failure instead of silently returning 0
- server: log warn when SetAudioCache fails after audio generation
- NewHybridStore: add explicit Ping() before EnsureCollections for fast-fail on bad credentials
- e2e fixture: replace single contentClient with directClient (plain HTTP)
for chapter/metadata/ranking + contentClient (Browserless) for urlClient
only — matches production wiring and is significantly faster
- server: add max_chars field to audio request body; truncates stripped text
to N runes before sending to Kokoro (used by e2e for quick TTS tests)
- fix: move RankingItem to scraper package to break novelfire→storage import
cycle; storage.RankingItem is now a type alias for backward compat
- fix: update stale New() call in novelfire integration test (missing args)
- fix: replace removed blob-ranking methods in storage integration test with
current per-item API (UpsertRankingItem/ListRankingItems/RankingLastUpdated)
- justfile: add test-e2e and e2e tasks
- Replace SetRanking/GetRanking/SetRankingPageHTML/GetRankingPageHTML blob methods
with WriteRankingItem/ReadRankingItems/RankingFreshEnough per-item operations
- Add 24h staleness gate in ScrapeRanking to skip re-scraping fresh data
- Add GET /api/ranking endpoint returning []RankingItem sorted by rank
- Remove RankingPageCacher interface and rankingCacheAdapter adapter
- Update integration tests to use new per-item upsert semantics
- Include e2e test suite (scraper/internal/e2e/)
- GET /api/browse fetches novelfire.net catalogue page and parses it with golang.org/x/net/html;
returns JSON {novels, page, hasNext} with per-novel slug/title/cover/rank/rating/chapters/url.
Supports page, genre, sort, status query params.
- GET /api/scrape/status returns {"running": bool} for polling job state from the UI
- Make overlay scroll within the viewport instead of being clipped (overflow-y-auto on backdrop, items-start alignment)
- Split summary text into paragraphs in the zoom overlay (newline-split with sentence-boundary fallback)
- Add Summary heading in overlay for context
- Remove old merge-poller JS; simplify generateAudio to use data.url directly (Kokoro backend rewrite cleanup)
- Extract scrape form and autocomplete JS from homeTmpl into new scrapeTmpl
- Add handleScrape GET handler serving /scrape with ranking autocomplete
- Register GET /scrape route in server.go
- Replace inline scrape form on home page with '+ Add' flat button in header
- handleHome no longer loads ranking items (only needed on /scrape)
- Split chapter TTS into up to 10 paragraph-aligned parts; part 0 is
generated synchronously so playback starts immediately, parts 1-9 and
the final merge happen in a background goroutine
- New routes: GET /ui/audio/{slug}/{n}/status (merge poll) and
GET /ui/audio-file/{slug}/{n}/part/{p} (serve individual part)
- JS polls status every 3 s and seamlessly swaps audio.src to the merged
file once ready, preserving playback position proportionally
- Home page scrape form replaced with a ranking-search autocomplete:
type a title/author to see matching ranking items (cover + metadata),
click or keyboard-select to inject the source URL, or paste a raw URL
directly; ranking data is embedded as JSON at page render time
- Add POST /ui/audio/{slug}/{n} endpoint: calls Kokoro-FastAPI server-side,
writes MP3 atomically to disk, deduplicates concurrent requests via
in-flight channel map, wraps route with 10-min TimeoutHandler
- Add GET /ui/audio-file/{slug}/{n} endpoint: serves cached MP3 with 1-day
cache headers
- Add AudioDir/AudioPath helpers to writer.go
- Rewrite JS TTS: remove all MSE/blob/stream code; use plain fetch to
generate endpoint then set audio.src to returned URL
- Add AbortController to generateAudio; abort on stop() and navigation
- Keep voice/speed controls disabled until setPlaying() fires
- Reset prefetchFired on voice/speed change
- Upgrade prefetch from fire-and-forget to promise chain updating queue badge
- Add sticky bottom audio queue panel: always-visible scrubber with timestamps,
expandable rows for Now/Next/Dbg with color-coded state badges
- Fix iOS Reader Mode: header->nav aria-hidden, audio outside nav, role=main,
visible h1 in content area, hover-only paragraph highlight
- Fix home screen Available books hidden: stop hiding cards in #books-grid
when they appear in Continue Reading; Available always shows all books
Two bugs caused the 'Refresh Rankings' button to silently fail in production:
1. ScrapeRanking was using the plain HTTP client (s.client) instead of the
browserless content client (s.urlClient). The /ranking page requires
JavaScript rendering, so a plain fetch returned HTML without any novel
entries. Now uses s.urlClient so the page is fully rendered before scraping.
2. handleRankingRefresh was synchronous, holding the HTTP connection open for
up to 60 s while scraping. Reverse proxies and HTMX timeouts closed the
connection before the scrape finished. Rewritten to the same async pattern
used for book scraping: POST /ranking/refresh returns immediately with a
polling badge; the browser polls GET /ui/ranking/status every 3 s; when
the goroutine finishes the status endpoint sends HX-Redirect to /ranking.
Chapter list:
- Book page now shows the first 50 chapters only; a 'Load more' button
(HTMX, GET /books/{slug}/chapters-page?page=N) appends the next page
without a full navigation, replacing itself with the next load-more
button or disappearing when all chapters are loaded.
Reading progress:
- Visiting a chapter saves {slug: chapterN} to localStorage under
'reading_progress'.
- The book page reads that key on load and, if a saved chapter exists,
highlights the saved chapter row with an amber dot and shows a
'Resume — Chapter N' button that navigates directly to it.
- Progress dots are also re-applied after each Load More via
hx-on::after-request.
- Add Kokoro-FastAPI TTS integration to the chapter reader UI:
- Browser-side MSE streaming with paragraph-level click-to-start
- Voice selector, speed slider, auto-next with prefetch of the next chapter
- New GET /ui/chapter-text endpoint that strips Markdown and serves plain text
- Add ranking page (novelfire /ranking scraper, WriteRanking/ReadRankingItems
in writer, GET /ranking + POST /ranking/refresh + GET /ranking/view routes)
with local-library annotation and one-click scrape buttons
- Add StrategyDirect (plain HTTP client) as a new browser strategy; the
default strategy is now 'direct' for chapter fetching and 'content'
for chapter-list URL retrieval (split via BROWSERLESS_URL_STRATEGY)
- Fix chapter numbering bug: numbers are now derived from the URL path
(/chapter-N) rather than list position, correcting newest-first ordering
- Add 'refresh <slug>' CLI sub-command to re-scrape a book from its saved
source_url without knowing the original URL
- Extend NovelScraper interface with RankingProvider (ScrapeRanking)
- Tune scraper timeouts: wait-for-selector reduced to 5 s, GotoOptions
timeout set to 60 s, content/scrape client defaults raised to 90 s
- Add cover extraction fix (figure.cover > img rather than bare img.cover)
- Add AGENTS.md and .aiignore for AI tooling context
- Add integration tests for browser client and novelfire scraper (build
tag: integration) and unit tests for chapterNumberFromURL and pagination
