feat: ranking page pagination, popular URL, and per-page HTML disk cache
Some checks failed
CI / Lint (push) Has been cancelled
CI / Test (push) Has been cancelled
CI / Build (push) Has been cancelled

- Switch ScrapeRanking to novelfire.net/genre-all/sort-popular URL and updated DOM selectors (div.novel-item, h3.novel-title, div.genres)
- Replace 5 hardcoded refresh buttons with dynamic 100-page paginator (smart ellipsis via rankingPageNums)
- Add RankingPageCacher interface and writer methods to cache raw HTML per page under static/books/_ranking_cache/page-N.html
- ScrapeRanking serves from disk cache on hit and writes to cache on miss, skipping Browserless round-trip
- Thread writer as PageCacher through novelfire.New and main.go
- Add TestScrapeRanking_CacheHit and TestScrapeRanking_CacheMiss tests
This commit is contained in:
Admin
2026-03-01 21:32:50 +05:00
parent 73869f01fa
commit 9cf94576d8
7 changed files with 334 additions and 103 deletions

View File

@@ -120,6 +120,16 @@ type RankingProvider interface {
ScrapeRanking(ctx context.Context, maxPages int) (<-chan BookMeta, <-chan error)
}
// RankingPageCacher persists and retrieves raw HTML for individual ranking pages.
// Implementations (e.g. writer.Writer) store files on disk so that a
// subsequent ScrapeRanking call can serve cached HTML without a network round-trip.
type RankingPageCacher interface {
// WriteRankingPageCache stores the raw HTML string for the given page number.
WriteRankingPageCache(page int, html string) error
// ReadRankingPageCache returns the cached HTML for page, or ("", nil) on a miss.
ReadRankingPageCache(page int) (string, error)
}
// NovelScraper is the full interface that a concrete novel source must implement.
// It composes all four provider interfaces.
type NovelScraper interface {