- New MinIO bucket 'libnovel-browse' (MINIO_BUCKET_BROWSE env) for storing
self-contained HTML snapshots of novelfire browse pages
- Store interface gains SaveBrowsePage / GetBrowsePage / BrowsePageKey methods
- handleBrowse is now cache-first: serves from MinIO snapshot when available,
then fires a background triggerBrowseSnapshot goroutine to populate cache
on live-fetch (de-duplicated, 90s timeout)
- New 'save-browse' CLI subcommand to bulk-capture pages via SingleFile CLI
- Dockerfile: downloads pinned single-file-x86_64-linux binary (v2.0.83),
adds gcompat + libstdc++ to Alpine runtime for glibc compatibility
- docker-compose: adds libnovel-browse bucket init and SINGLEFILE_PATH env
- .gitignore: exclude scraper/scraper build artifact
- ui/Dockerfile: copy package-lock.json and run npm ci --omit=dev in the
runtime stage so marked (and other runtime deps) are available to
adapter-node at startup — fixes ERR_MODULE_NOT_FOUND for 'marked'
- storage: add ReindexChapters to Store interface and HybridStore — walks
MinIO objects for a slug, reads chapter titles, upserts chapters_idx
- server: add POST /api/reindex/{slug} to rebuild chapters_idx from MinIO
runAsync creates a scraping_tasks record on job start, flushes progress
counters via OnProgress, and finalizes status (done/failed/cancelled) on
completion. Adds GET /api/scrape/tasks to list all historical jobs.
Also fixes relative cover URLs in parseBrowsePage.
- Inject *slog.Logger into HybridStore, PocketBaseStore, and pbClient
- Fix credential defaults in main.go (changeme123 / admin) to match docker-compose
- listOne/listAll/upsert/deleteWhere now return errors on non-2xx HTTP status
- WriteChapter: log warn instead of discarding UpsertChapterIdx error
- MetadataMtime, GetAudioCache, GetProgress: log warn on PocketBase failures
- EnsureCollections: log info/debug/warn per outcome instead of _ = err
- CountChapterIdx: log warn on failure instead of silently returning 0
- server: log warn when SetAudioCache fails after audio generation
- NewHybridStore: add explicit Ping() before EnsureCollections for fast-fail on bad credentials
- e2e fixture: replace single contentClient with directClient (plain HTTP)
for chapter/metadata/ranking + contentClient (Browserless) for urlClient
only — matches production wiring and is significantly faster
- server: add max_chars field to audio request body; truncates stripped text
to N runes before sending to Kokoro (used by e2e for quick TTS tests)
- fix: move RankingItem to scraper package to break novelfire→storage import
cycle; storage.RankingItem is now a type alias for backward compat
- fix: update stale New() call in novelfire integration test (missing args)
- fix: replace removed blob-ranking methods in storage integration test with
current per-item API (UpsertRankingItem/ListRankingItems/RankingLastUpdated)
- justfile: add test-e2e and e2e tasks
- Replace SetRanking/GetRanking/SetRankingPageHTML/GetRankingPageHTML blob methods
with WriteRankingItem/ReadRankingItems/RankingFreshEnough per-item operations
- Add 24h staleness gate in ScrapeRanking to skip re-scraping fresh data
- Add GET /api/ranking endpoint returning []RankingItem sorted by rank
- Remove RankingPageCacher interface and rankingCacheAdapter adapter
- Update integration tests to use new per-item upsert semantics
- Include e2e test suite (scraper/internal/e2e/)
- GET /api/browse fetches novelfire.net catalogue page and parses it with golang.org/x/net/html;
returns JSON {novels, page, hasNext} with per-novel slug/title/cover/rank/rating/chapters/url.
Supports page, genre, sort, status query params.
- GET /api/scrape/status returns {"running": bool} for polling job state from the UI
- Make overlay scroll within the viewport instead of being clipped (overflow-y-auto on backdrop, items-start alignment)
- Split summary text into paragraphs in the zoom overlay (newline-split with sentence-boundary fallback)
- Add Summary heading in overlay for context
- Remove old merge-poller JS; simplify generateAudio to use data.url directly (Kokoro backend rewrite cleanup)
- Extract scrape form and autocomplete JS from homeTmpl into new scrapeTmpl
- Add handleScrape GET handler serving /scrape with ranking autocomplete
- Register GET /scrape route in server.go
- Replace inline scrape form on home page with '+ Add' flat button in header
- handleHome no longer loads ranking items (only needed on /scrape)
- Split chapter TTS into up to 10 paragraph-aligned parts; part 0 is
generated synchronously so playback starts immediately, parts 1-9 and
the final merge happen in a background goroutine
- New routes: GET /ui/audio/{slug}/{n}/status (merge poll) and
GET /ui/audio-file/{slug}/{n}/part/{p} (serve individual part)
- JS polls status every 3 s and seamlessly swaps audio.src to the merged
file once ready, preserving playback position proportionally
- Home page scrape form replaced with a ranking-search autocomplete:
type a title/author to see matching ranking items (cover + metadata),
click or keyboard-select to inject the source URL, or paste a raw URL
directly; ranking data is embedded as JSON at page render time
- Add POST /ui/audio/{slug}/{n} endpoint: calls Kokoro-FastAPI server-side,
writes MP3 atomically to disk, deduplicates concurrent requests via
in-flight channel map, wraps route with 10-min TimeoutHandler
- Add GET /ui/audio-file/{slug}/{n} endpoint: serves cached MP3 with 1-day
cache headers
- Add AudioDir/AudioPath helpers to writer.go
- Rewrite JS TTS: remove all MSE/blob/stream code; use plain fetch to
generate endpoint then set audio.src to returned URL
- Add AbortController to generateAudio; abort on stop() and navigation
- Keep voice/speed controls disabled until setPlaying() fires
- Reset prefetchFired on voice/speed change
- Upgrade prefetch from fire-and-forget to promise chain updating queue badge
- Add sticky bottom audio queue panel: always-visible scrubber with timestamps,
expandable rows for Now/Next/Dbg with color-coded state badges
- Fix iOS Reader Mode: header->nav aria-hidden, audio outside nav, role=main,
visible h1 in content area, hover-only paragraph highlight
- Fix home screen Available books hidden: stop hiding cards in #books-grid
when they appear in Continue Reading; Available always shows all books
Two bugs caused the 'Refresh Rankings' button to silently fail in production:
1. ScrapeRanking was using the plain HTTP client (s.client) instead of the
browserless content client (s.urlClient). The /ranking page requires
JavaScript rendering, so a plain fetch returned HTML without any novel
entries. Now uses s.urlClient so the page is fully rendered before scraping.
2. handleRankingRefresh was synchronous, holding the HTTP connection open for
up to 60 s while scraping. Reverse proxies and HTMX timeouts closed the
connection before the scrape finished. Rewritten to the same async pattern
used for book scraping: POST /ranking/refresh returns immediately with a
polling badge; the browser polls GET /ui/ranking/status every 3 s; when
the goroutine finishes the status endpoint sends HX-Redirect to /ranking.
Chapter list:
- Book page now shows the first 50 chapters only; a 'Load more' button
(HTMX, GET /books/{slug}/chapters-page?page=N) appends the next page
without a full navigation, replacing itself with the next load-more
button or disappearing when all chapters are loaded.
Reading progress:
- Visiting a chapter saves {slug: chapterN} to localStorage under
'reading_progress'.
- The book page reads that key on load and, if a saved chapter exists,
highlights the saved chapter row with an amber dot and shows a
'Resume — Chapter N' button that navigates directly to it.
- Progress dots are also re-applied after each Load More via
hx-on::after-request.
- Add Kokoro-FastAPI TTS integration to the chapter reader UI:
- Browser-side MSE streaming with paragraph-level click-to-start
- Voice selector, speed slider, auto-next with prefetch of the next chapter
- New GET /ui/chapter-text endpoint that strips Markdown and serves plain text
- Add ranking page (novelfire /ranking scraper, WriteRanking/ReadRankingItems
in writer, GET /ranking + POST /ranking/refresh + GET /ranking/view routes)
with local-library annotation and one-click scrape buttons
- Add StrategyDirect (plain HTTP client) as a new browser strategy; the
default strategy is now 'direct' for chapter fetching and 'content'
for chapter-list URL retrieval (split via BROWSERLESS_URL_STRATEGY)
- Fix chapter numbering bug: numbers are now derived from the URL path
(/chapter-N) rather than list position, correcting newest-first ordering
- Add 'refresh <slug>' CLI sub-command to re-scrape a book from its saved
source_url without knowing the original URL
- Extend NovelScraper interface with RankingProvider (ScrapeRanking)
- Tune scraper timeouts: wait-for-selector reduced to 5 s, GotoOptions
timeout set to 60 s, content/scrape client defaults raised to 90 s
- Add cover extraction fix (figure.cover > img rather than bare img.cover)
- Add AGENTS.md and .aiignore for AI tooling context
- Add integration tests for browser client and novelfire scraper (build
tag: integration) and unit tests for chapterNumberFromURL and pagination
