novelfire.net chapter content is server-rendered, so Browserless is not
needed. Add a dedicated chapterClient (always StrategyDirect) to Scraper
and use it in ScrapeChapterText, removing the now-irrelevant WaitFor /
RejectResourceTypes / GotoOptions fields from the ContentRequest.
- Replace SetRanking/GetRanking/SetRankingPageHTML/GetRankingPageHTML blob methods
with WriteRankingItem/ReadRankingItems/RankingFreshEnough per-item operations
- Add 24h staleness gate in ScrapeRanking to skip re-scraping fresh data
- Add GET /api/ranking endpoint returning []RankingItem sorted by rank
- Remove RankingPageCacher interface and rankingCacheAdapter adapter
- Update integration tests to use new per-item upsert semantics
- Include e2e test suite (scraper/internal/e2e/)
The previous selectors were based on a hypothetical structure that does not
match the actual site. The real novelfire.net popular listing uses:
<li class="novel-item"> (not <div>)
<a href="/book/slug" title="Title">
<figure class="novel-cover"><img data-src="/path.jpg"></figure>
<h4 class="novel-title text2row">Title</h4>
</a>
</li>
And pagination uses <a rel="next"> (not <a class="next">).
Changes:
- ScrapeRanking: use li.novel-item, h4.novel-title, figure.novel-cover,
img[data-src]; strip base64 placeholder covers
- hasNextPageLink(): new helper walking all <a> nodes for rel="next"
- Import golang.org/x/net/html for Node.Attr access
- Test fixtures rewritten to match real structure (li/h4/rel=next)
- Status and genres removed from ranking items (not present on listing page)
Verified end-to-end: ranking.json written with 24 items on first fetch
- Switch ScrapeRanking to novelfire.net/genre-all/sort-popular URL and updated DOM selectors (div.novel-item, h3.novel-title, div.genres)
- Replace 5 hardcoded refresh buttons with dynamic 100-page paginator (smart ellipsis via rankingPageNums)
- Add RankingPageCacher interface and writer methods to cache raw HTML per page under static/books/_ranking_cache/page-N.html
- ScrapeRanking serves from disk cache on hit and writes to cache on miss, skipping Browserless round-trip
- Thread writer as PageCacher through novelfire.New and main.go
- Add TestScrapeRanking_CacheHit and TestScrapeRanking_CacheMiss tests
- Fix channel drain goroutine deadlock in handleRankingRefresh: replace
'for { ... continue ... if nil break }' with 'for A != nil || B != nil'
so the select never blocks on two nil channels
- Switch ScrapeRanking from urlClient (Browserless) to client (direct HTTP)
since novelfire.net/ranking is fully server-rendered — no JS needed
- Add ranking unit tests (single page, multi-page, empty page, write round-trip)
- Add .gitea/workflows/ci.yaml: lint, test, build jobs with commented-out
Docker image push step for when runner has Docker available
