kamil/libnovel
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 162 Wiki Activity

fix(scraper): add Brotli decompression to HTTP client
Some checks failed
CI / Scraper / Lint (push) Failing after 29s
Details
CI / Scraper / Lint (pull_request) Failing after 29s
Details
CI / Scraper / Test (push) Failing after 38s
Details
CI / Scraper / Docker Push (push) Has been skipped
Details
CI / UI / Build (pull_request) Successful in 47s
Details
CI / UI / Docker Push (pull_request) Has been skipped
Details
CI / Scraper / Test (pull_request) Successful in 54s
Details
CI / Scraper / Docker Push (pull_request) Has been skipped
Details
iOS CI / Build (pull_request) Successful in 3m35s
Details
iOS CI / Test (pull_request) Successful in 5m47s
Details

Browse Source 
novelfire.net responds with Content-Encoding: br when the scraper
advertises 'gzip, deflate, br'. The client only handled gzip, so
Brotli-compressed bytes were fed raw into the HTML parser producing
garbage — empty titles, zero chapters, and selector failures.

Added github.com/andybalholm/brotli and wired it into GetContent
alongside the existing gzip path.
This commit is contained in:
Admin
2026-03-20 11:19:28 +05:00
parent 1642434a79
commit 1a49cb5e75
3 changed files with 9 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
scraper/go.sum
View File

@@ -1,5 +1,7 @@
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=