5b27d501af
- Caddy: custom image with caddy-ratelimit plugin, security headers (X-Frame-Options, HSTS, CSP-adjacent, etc.), per-IP rate limiting on auth/scrape/global zones, static error pages (502/503/504), fix routing to remove /api/scrape/* and /api/chapter-text-preview/* direct-to-backend (were bypassing SvelteKit auth middleware) - docker-compose: Caddy build context + error volume, Watchtower service (label-enable mode, 5 min poll), watchtower labels on backend/runner/ui - Scraper: ScrapeChapterList uses retryGet (9 attempts, Retry-After backoff) to fix 429-induced chapter list failures; upTo param stops pagination early for range scrapes - UI: Browse→Catalogue rename (routes, API, links), admin scrape page Continue/Retry buttons, +error.svelte branded error page, type cleanup (removed dead exports, added BookPreviewMeta/BookPreviewResponse to scraper.ts) - Meilisearch: meta_updated field, sort=update fix, facet distribution - Docs: reorganise into docs/d2/ and docs/mermaid/ subdirectories, update all diagrams to reflect Caddy/Watchtower/routing changes, add api-routing.d2 ownership map with auth-level colour coding, regenerate SVGs
61 lines
2.0 KiB
Go
61 lines
2.0 KiB
Go
// Package scraper defines the NovelScraper interface and its sub-interfaces.
|
|
// Domain types live in internal/domain — this package only defines the scraping
|
|
// contract so that novelfire and any future scrapers can be swapped freely.
|
|
package scraper
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/libnovel/backend/internal/domain"
|
|
)
|
|
|
|
// CatalogueProvider can enumerate every novel available on a source site.
|
|
type CatalogueProvider interface {
|
|
ScrapeCatalogue(ctx context.Context) (<-chan domain.CatalogueEntry, <-chan error)
|
|
}
|
|
|
|
// MetadataProvider can extract structured book metadata from a novel's landing page.
|
|
type MetadataProvider interface {
|
|
ScrapeMetadata(ctx context.Context, bookURL string) (domain.BookMeta, error)
|
|
}
|
|
|
|
// ChapterListProvider can enumerate all chapters of a book.
|
|
// upTo > 0 stops pagination once at least upTo chapter numbers have been
|
|
// collected (early-exit optimisation for range scrapes). upTo == 0 fetches all pages.
|
|
type ChapterListProvider interface {
|
|
ScrapeChapterList(ctx context.Context, bookURL string, upTo int) ([]domain.ChapterRef, error)
|
|
}
|
|
|
|
// ChapterTextProvider can extract the readable text from a single chapter page.
|
|
type ChapterTextProvider interface {
|
|
ScrapeChapterText(ctx context.Context, ref domain.ChapterRef) (domain.Chapter, error)
|
|
}
|
|
|
|
// RankingProvider can enumerate novels from a ranking page.
|
|
type RankingProvider interface {
|
|
// ScrapeRanking pages through up to maxPages ranking pages.
|
|
// maxPages <= 0 means all pages.
|
|
ScrapeRanking(ctx context.Context, maxPages int) (<-chan domain.BookMeta, <-chan error)
|
|
}
|
|
|
|
// NovelScraper is the full interface a concrete novel source must implement.
|
|
type NovelScraper interface {
|
|
CatalogueProvider
|
|
MetadataProvider
|
|
ChapterListProvider
|
|
ChapterTextProvider
|
|
RankingProvider
|
|
|
|
// SourceName returns the human-readable name of this scraper, e.g. "novelfire.net".
|
|
SourceName() string
|
|
}
|
|
|
|
// Selector describes how to locate an element in an HTML document.
|
|
type Selector struct {
|
|
Tag string
|
|
Class string
|
|
ID string
|
|
Attr string
|
|
Multiple bool
|
|
}
|