kamil/libnovel
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 162 Wiki Activity

fix: make ranking refresh async and use JS-rendered client for ScrapeRanking

Browse Source 
Two bugs caused the 'Refresh Rankings' button to silently fail in production:

1. ScrapeRanking was using the plain HTTP client (s.client) instead of the
   browserless content client (s.urlClient). The /ranking page requires
   JavaScript rendering, so a plain fetch returned HTML without any novel
   entries. Now uses s.urlClient so the page is fully rendered before scraping.

2. handleRankingRefresh was synchronous, holding the HTTP connection open for
   up to 60 s while scraping. Reverse proxies and HTMX timeouts closed the
   connection before the scrape finished. Rewritten to the same async pattern
   used for book scraping: POST /ranking/refresh returns immediately with a
   polling badge; the browser polls GET /ui/ranking/status every 3 s; when
   the goroutine finishes the status endpoint sends HX-Redirect to /ranking.
This commit is contained in:
Admin
2026-03-01 16:52:03 +05:00
parent bcdef02997
commit e9f880f7f7
3 changed files with 111 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
scraper/internal/novelfire/scraper.go
View File

@@ -385,23 +385,14 @@ func (s *Scraper) ScrapeRanking(ctx context.Context) (<-chan scraper.BookMeta, <
s.log.Info("scraping ranking page", "url", pageURL)
// Use WaitFor only for browser-based strategies
var raw string
var err error
if s.client.Strategy() == browser.StrategyDirect {
raw, err = s.client.GetContent(ctx, browser.ContentRequest{
URL: pageURL,
RejectResourceTypes: rejectResourceTypes,
})
} else {
raw, err = s.client.GetContent(ctx, browser.ContentRequest{
URL: pageURL,
WaitFor: &browser.WaitForSelector{Selector: ".rank-novels", Timeout: 30000},
RejectResourceTypes: rejectResourceTypes,
GotoOptions: &browser.GotoOptions{Timeout: 60000},
BestAttempt: true,
})
}
// Always use the urlClient for the ranking page — it requires JS rendering.
raw, err := s.urlClient.GetContent(ctx, browser.ContentRequest{
URL: pageURL,
WaitFor: &browser.WaitForSelector{Selector: ".rank-novels", Timeout: 30000},
RejectResourceTypes: rejectResourceTypes,
GotoOptions: &browser.GotoOptions{Timeout: 60000},
BestAttempt: true,
})
if err != nil {
s.log.Debug("ranking page fetch failed", "url", pageURL, "err", err)
errs <- fmt.Errorf("ranking page: %w", err)