feat: ranking page pagination, popular URL, and per-page HTML disk cache
- Switch ScrapeRanking to novelfire.net/genre-all/sort-popular URL and updated DOM selectors (div.novel-item, h3.novel-title, div.genres) - Replace 5 hardcoded refresh buttons with dynamic 100-page paginator (smart ellipsis via rankingPageNums) - Add RankingPageCacher interface and writer methods to cache raw HTML per page under static/books/_ranking_cache/page-N.html - ScrapeRanking serves from disk cache on hit and writes to cache on miss, skipping Browserless round-trip - Thread writer as PageCacher through novelfire.New and main.go - Add TestScrapeRanking_CacheHit and TestScrapeRanking_CacheMiss tests
This commit is contained in:
@@ -120,6 +120,16 @@ type RankingProvider interface {
|
||||
ScrapeRanking(ctx context.Context, maxPages int) (<-chan BookMeta, <-chan error)
|
||||
}
|
||||
|
||||
// RankingPageCacher persists and retrieves raw HTML for individual ranking pages.
|
||||
// Implementations (e.g. writer.Writer) store files on disk so that a
|
||||
// subsequent ScrapeRanking call can serve cached HTML without a network round-trip.
|
||||
type RankingPageCacher interface {
|
||||
// WriteRankingPageCache stores the raw HTML string for the given page number.
|
||||
WriteRankingPageCache(page int, html string) error
|
||||
// ReadRankingPageCache returns the cached HTML for page, or ("", nil) on a miss.
|
||||
ReadRankingPageCache(page int) (string, error)
|
||||
}
|
||||
|
||||
// NovelScraper is the full interface that a concrete novel source must implement.
|
||||
// It composes all four provider interfaces.
|
||||
type NovelScraper interface {
|
||||
|
||||
Reference in New Issue
Block a user