feat: paginated ranking scrape with lazy page fetching

- ScrapeRanking now accepts a maxPages int parameter (0 = all pages).
  Each page is fetched strictly sequentially; the next page is only
  requested after every entry from the current page has been sent,
  so there is no pre-fetching or look-ahead.
  Pagination stops automatically when no next-page link is present
  or when the rank-novels container is absent/empty.

- The ranking URL pattern follows the existing catalogue convention:
  /ranking?page=N (next-page link detection as the stop condition).

- Server: handleRankingRefresh reads an optional 'pages' form field
  and passes it to ScrapeRanking. Timeout scales at 90 s/page.

- UI: Refresh Rankings button is now a small form with a numeric
  'Pages' input (default 1), letting the user choose how many pages
  to pull in one refresh without touching the server config.
This commit is contained in:
Admin
2026-03-01 16:54:52 +05:00
parent e9f880f7f7
commit 1469e49190
3 changed files with 61 additions and 20 deletions

View File

@@ -112,9 +112,12 @@ type ChapterTextProvider interface {
// RankingProvider can enumerate novels from a ranking page.
type RankingProvider interface {
// ScrapeRanking pages through the ranking list, sending BookMeta values
// (with basic info like title, cover, genres, status, sourceURL) to the returned channel.
ScrapeRanking(ctx context.Context) (<-chan BookMeta, <-chan error)
// ScrapeRanking pages through up to maxPages ranking pages, sending BookMeta
// values (with basic info like title, cover, genres, status, sourceURL) to
// the returned channel. Pages are fetched sequentially and lazily: the next
// page is only requested once all entries from the current page have been
// sent. maxPages <= 0 means "all pages".
ScrapeRanking(ctx context.Context, maxPages int) (<-chan BookMeta, <-chan error)
}
// NovelScraper is the full interface that a concrete novel source must implement.