The previous selectors were based on a hypothetical structure that does not
match the actual site. The real novelfire.net popular listing uses:
<li class="novel-item"> (not <div>)
<a href="/book/slug" title="Title">
<figure class="novel-cover"><img data-src="/path.jpg"></figure>
<h4 class="novel-title text2row">Title</h4>
</a>
</li>
And pagination uses <a rel="next"> (not <a class="next">).
Changes:
- ScrapeRanking: use li.novel-item, h4.novel-title, figure.novel-cover,
img[data-src]; strip base64 placeholder covers
- hasNextPageLink(): new helper walking all <a> nodes for rel="next"
- Import golang.org/x/net/html for Node.Attr access
- Test fixtures rewritten to match real structure (li/h4/rel=next)
- Status and genres removed from ranking items (not present on listing page)
Verified end-to-end: ranking.json written with 24 items on first fetch
- Switch ScrapeRanking to novelfire.net/genre-all/sort-popular URL and updated DOM selectors (div.novel-item, h3.novel-title, div.genres)
- Replace 5 hardcoded refresh buttons with dynamic 100-page paginator (smart ellipsis via rankingPageNums)
- Add RankingPageCacher interface and writer methods to cache raw HTML per page under static/books/_ranking_cache/page-N.html
- ScrapeRanking serves from disk cache on hit and writes to cache on miss, skipping Browserless round-trip
- Thread writer as PageCacher through novelfire.New and main.go
- Add TestScrapeRanking_CacheHit and TestScrapeRanking_CacheMiss tests
- Fix channel drain goroutine deadlock in handleRankingRefresh: replace
'for { ... continue ... if nil break }' with 'for A != nil || B != nil'
so the select never blocks on two nil channels
- Switch ScrapeRanking from urlClient (Browserless) to client (direct HTTP)
since novelfire.net/ranking is fully server-rendered — no JS needed
- Add ranking unit tests (single page, multi-page, empty page, write round-trip)
- Add .gitea/workflows/ci.yaml: lint, test, build jobs with commented-out
Docker image push step for when runner has Docker available
- ScrapeRanking now accepts a maxPages int parameter (0 = all pages).
Each page is fetched strictly sequentially; the next page is only
requested after every entry from the current page has been sent,
so there is no pre-fetching or look-ahead.
Pagination stops automatically when no next-page link is present
or when the rank-novels container is absent/empty.
- The ranking URL pattern follows the existing catalogue convention:
/ranking?page=N (next-page link detection as the stop condition).
- Server: handleRankingRefresh reads an optional 'pages' form field
and passes it to ScrapeRanking. Timeout scales at 90 s/page.
- UI: Refresh Rankings button is now a small form with a numeric
'Pages' input (default 1), letting the user choose how many pages
to pull in one refresh without touching the server config.
Two bugs caused the 'Refresh Rankings' button to silently fail in production:
1. ScrapeRanking was using the plain HTTP client (s.client) instead of the
browserless content client (s.urlClient). The /ranking page requires
JavaScript rendering, so a plain fetch returned HTML without any novel
entries. Now uses s.urlClient so the page is fully rendered before scraping.
2. handleRankingRefresh was synchronous, holding the HTTP connection open for
up to 60 s while scraping. Reverse proxies and HTMX timeouts closed the
connection before the scrape finished. Rewritten to the same async pattern
used for book scraping: POST /ranking/refresh returns immediately with a
polling badge; the browser polls GET /ui/ranking/status every 3 s; when
the goroutine finishes the status endpoint sends HX-Redirect to /ranking.
- Add Kokoro-FastAPI TTS integration to the chapter reader UI:
- Browser-side MSE streaming with paragraph-level click-to-start
- Voice selector, speed slider, auto-next with prefetch of the next chapter
- New GET /ui/chapter-text endpoint that strips Markdown and serves plain text
- Add ranking page (novelfire /ranking scraper, WriteRanking/ReadRankingItems
in writer, GET /ranking + POST /ranking/refresh + GET /ranking/view routes)
with local-library annotation and one-click scrape buttons
- Add StrategyDirect (plain HTTP client) as a new browser strategy; the
default strategy is now 'direct' for chapter fetching and 'content'
for chapter-list URL retrieval (split via BROWSERLESS_URL_STRATEGY)
- Fix chapter numbering bug: numbers are now derived from the URL path
(/chapter-N) rather than list position, correcting newest-first ordering
- Add 'refresh <slug>' CLI sub-command to re-scrape a book from its saved
source_url without knowing the original URL
- Extend NovelScraper interface with RankingProvider (ScrapeRanking)
- Tune scraper timeouts: wait-for-selector reduced to 5 s, GotoOptions
timeout set to 60 s, content/scrape client defaults raised to 90 s
- Add cover extraction fix (figure.cover > img rather than bare img.cover)
- Add AGENTS.md and .aiignore for AI tooling context
- Add integration tests for browser client and novelfire scraper (build
tag: integration) and unit tests for chapterNumberFromURL and pagination
