feat: paginated ranking scrape with lazy page fetching
- ScrapeRanking now accepts a maxPages int parameter (0 = all pages). Each page is fetched strictly sequentially; the next page is only requested after every entry from the current page has been sent, so there is no pre-fetching or look-ahead. Pagination stops automatically when no next-page link is present or when the rank-novels container is absent/empty. - The ranking URL pattern follows the existing catalogue convention: /ranking?page=N (next-page link detection as the stop condition). - Server: handleRankingRefresh reads an optional 'pages' form field and passes it to ScrapeRanking. Timeout scales at 90 s/page. - UI: Refresh Rankings button is now a small form with a numeric 'Pages' input (default 1), letting the user choose how many pages to pull in one refresh without touching the server config.
This commit is contained in:
@@ -112,9 +112,12 @@ type ChapterTextProvider interface {
|
||||
|
||||
// RankingProvider can enumerate novels from a ranking page.
|
||||
type RankingProvider interface {
|
||||
// ScrapeRanking pages through the ranking list, sending BookMeta values
|
||||
// (with basic info like title, cover, genres, status, sourceURL) to the returned channel.
|
||||
ScrapeRanking(ctx context.Context) (<-chan BookMeta, <-chan error)
|
||||
// ScrapeRanking pages through up to maxPages ranking pages, sending BookMeta
|
||||
// values (with basic info like title, cover, genres, status, sourceURL) to
|
||||
// the returned channel. Pages are fetched sequentially and lazily: the next
|
||||
// page is only requested once all entries from the current page have been
|
||||
// sent. maxPages <= 0 means "all pages".
|
||||
ScrapeRanking(ctx context.Context, maxPages int) (<-chan BookMeta, <-chan error)
|
||||
}
|
||||
|
||||
// NovelScraper is the full interface that a concrete novel source must implement.
|
||||
|
||||
Reference in New Issue
Block a user