Admin
1a49cb5e75
fix(scraper): add Brotli decompression to HTTP client
...
CI / Scraper / Lint (push) Failing after 29s
CI / Scraper / Lint (pull_request) Failing after 29s
CI / Scraper / Test (push) Failing after 38s
CI / Scraper / Docker Push (push) Has been skipped
CI / UI / Build (pull_request) Successful in 47s
CI / UI / Docker Push (pull_request) Has been skipped
CI / Scraper / Test (pull_request) Successful in 54s
CI / Scraper / Docker Push (pull_request) Has been skipped
iOS CI / Build (pull_request) Successful in 3m35s
iOS CI / Test (pull_request) Successful in 5m47s
novelfire.net responds with Content-Encoding: br when the scraper
advertises 'gzip, deflate, br'. The client only handled gzip, so
Brotli-compressed bytes were fed raw into the HTML parser producing
garbage — empty titles, zero chapters, and selector failures.
Added github.com/andybalholm/brotli and wired it into GetContent
alongside the existing gzip path.
2026-03-20 11:20:50 +05:00
Admin
fb6b364382
refactor: audit, split server.go, add unit tests, and fix latent bugs
...
- Remove dead code: browser cdp/content_scrape strategies, writer package,
printUsage, downloadAndStoreCoverCLI in main.go
- Fix bugs: defer-in-loop in pocketbase deleteWhere, listAll() pagination
hard cap removed, splitChapterTitle off-by-one in date extraction
- Split server.go (~1700 lines) into focused handler files:
handlers_audio, handlers_browse, handlers_progress, handlers_ranking,
handlers_scrape
- Export htmlutil.AttrVal/TextContent/ResolveURL; add storage/coverutil.go
to consolidate duplicate helpers
- Flatten deeply nested conditionals: voices() early-return guards,
ScrapeCatalogue next-link double attr scan, chapterNumberFromKey dead
strings.Cut line, splitChapterTitle double-nested unit/suffix loop
- Add unit tests: htmlutil (9 funcs), novelfire ScrapeMetadata (3 cases),
orchestrator Run (5 cases), storage chapterNumberFromKey/splitChapterTitle
(22 cases); all pass with go build/vet/test clean
2026-03-04 22:14:23 +05:00
