The previous ensure_fields helper used python3 which is not available in
alpine:3.19, causing exit 127 in the cloud init container.
Replaced with ensure_field (per-field, no python/jq) that uses only
busybox sh, wget, sed, and awk. Also fixed the HTTP status grep pattern
in create_collection and ensure_field to match only the response header
line (^ HTTP/) instead of the wget error line, eliminating the spurious
'unexpected status server' warnings.
The progress collection was created before user_id was added to the schema,
causing all user-keyed progress queries to return 400. Add an ensure_fields
helper that idempotently patches existing collections with missing fields,
and run it for progress.user_id on every init.
Replace the single-button placeholder with a full homepage:
- Stats bar: total books, total chapters, books in progress
- Continue Reading grid (up to 6): links directly to last chapter read
- Recently Updated grid (up to 6): recent books not already in progress
- Empty state with Discover Novels CTA when library is empty
New pocketbase.ts helpers: recentlyAddedBooks(), getHomeStats(),
listN(), countCollection().
The pre-compiled single-file binary requires glibc symbols (__res_init)
that Alpine's gcompat shim does not provide, causing exit status 127.
Switch runtime base to node:22-alpine and install single-file-cli via npm.
Also add .dockerignore to exclude bin/ and static/ from build context.
- New MinIO bucket 'libnovel-browse' (MINIO_BUCKET_BROWSE env) for storing
self-contained HTML snapshots of novelfire browse pages
- Store interface gains SaveBrowsePage / GetBrowsePage / BrowsePageKey methods
- handleBrowse is now cache-first: serves from MinIO snapshot when available,
then fires a background triggerBrowseSnapshot goroutine to populate cache
on live-fetch (de-duplicated, 90s timeout)
- New 'save-browse' CLI subcommand to bulk-capture pages via SingleFile CLI
- Dockerfile: downloads pinned single-file-x86_64-linux binary (v2.0.83),
adds gcompat + libstdc++ to Alpine runtime for glibc compatibility
- docker-compose: adds libnovel-browse bucket init and SINGLEFILE_PATH env
- .gitignore: exclude scraper/scraper build artifact
The /ranking route is removed. /browse becomes the Discover page with:
- Sort=Ranking option: fetches from /api/ranking (richer metadata — author,
genres, source_url) and renders the full pre-computed ranked list
- Sort=Popular/New/Updated: fetches from /api/browse as before, paginated
- Grid / List view toggle: grid is the default for browse; list auto-selects
for ranking and gives the information-dense ranked layout (was /ranking)
- Admin Refresh catalogue button (moved from /ranking) and per-novel Scrape
button work in both views
- Genre and status filters are disabled (visually dimmed) when sort=rank
since the ranking endpoint does not support per-page filtering
- Nav: 'Browse' renamed to 'Discover', 'Ranking' link removed
Add optional user_id field to the progress collection. When a user is
authenticated, progress is keyed by user_id instead of session_id, making
it portable across devices and browsers. Anonymous reading still works via
session_id as before.
On login or registration, any progress accumulated under the anonymous
session is merged into the user's account (most-recent timestamp wins),
so chapters read before logging in are not lost.
- scripts/pb-init.sh: add user_id field to progress collection schema
- scraper/internal/storage/pocketbase.go: add user_id to EnsureCollections
- ui/src/lib/server/pocketbase.ts: Progress interface gains user_id;
getProgress/allProgress/setProgress accept optional userId and query/write
by user_id when present; add mergeSessionProgress() helper
- ui/src/routes/login/+page.server.ts: call mergeSessionProgress after
successful login and registration (fire-and-forget, non-fatal)
- ui/src/routes/api/progress/+server.ts: pass locals.user?.id to setProgress
- ui/src/routes/books/+page.server.ts: pass locals.user?.id to allProgress
- ui/src/routes/books/[slug]/+page.server.ts: pass locals.user?.id to getProgress
Add a second MinIO client (pub) initialized with MINIO_PUBLIC_ENDPOINT so
presigned audio URLs are signed against the public hostname from the start,
rather than signed internally and then rewritten. This avoids AWS4 signature
mismatch (SignatureDoesNotMatch 403) that occurred when the signed host was
substituted after signing.
- storage/minio.go: add PublicEndpoint/PublicUseSSL to MinioConfig; add pub
client field; NewMinioClient creates pub client when public endpoint differs;
PresignAudio uses pub, PresignChapter keeps internal client
- cmd/scraper/main.go: wire MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL env vars
- docker-compose.yml: expose MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL to scraper service
- ui/src/lib/server/minio.ts: remove rewriteHost() call from presignAudio
The SvelteKit server fetches chapter markdown server-side using the presigned
URL. rewriteHost() was replacing the internal minio:9000 host with the public
PUBLIC_MINIO_PUBLIC_URL (localhost:9000), which is unreachable from inside
Docker, causing MinIO 403/connection errors.
presignChapter now takes an optional rewrite=false parameter — the server-side
load function gets the raw internal URL, while any future browser-facing use
can pass rewrite=true to get the public-facing URL.
- ui/Dockerfile: copy package-lock.json and run npm ci --omit=dev in the
runtime stage so marked (and other runtime deps) are available to
adapter-node at startup — fixes ERR_MODULE_NOT_FOUND for 'marked'
- storage: add ReindexChapters to Store interface and HybridStore — walks
MinIO objects for a slug, reads chapter titles, upserts chapters_idx
- server: add POST /api/reindex/{slug} to rebuild chapters_idx from MinIO
Replace selected={data.x === opt.value} on individual <option> elements
with value={data.x} on the <select> — the idiomatic Svelte approach that
ensures correct hydration and form submission of the current filter values.
Adds /ranking route showing the cached ranking list with cover, title,
author, status, and genres. Admin users get a Refresh button that
triggers a full catalogue scrape. Adds Ranking to the nav.
runAsync creates a scraping_tasks record on job start, flushes progress
counters via OnProgress, and finalizes status (done/failed/cancelled) on
completion. Adds GET /api/scrape/tasks to list all historical jobs.
Also fixes relative cover URLs in parseBrowsePage.
Adds atomic counters for books_found, chapters_scraped, chapters_skipped,
and errors. The new OnProgress callback fires after each counter update
and once more at the end of Run, giving callers a live progress feed.
Adds a scraping_tasks PocketBase collection with fields for kind, status,
progress counters, timestamps, and error info. Exposes CreateScrapeTask,
UpdateScrapeTask, and ListScrapeTasks on the Store interface with
implementations in HybridStore and PocketBaseStore.
- Inject *slog.Logger into HybridStore, PocketBaseStore, and pbClient
- Fix credential defaults in main.go (changeme123 / admin) to match docker-compose
- listOne/listAll/upsert/deleteWhere now return errors on non-2xx HTTP status
- WriteChapter: log warn instead of discarding UpsertChapterIdx error
- MetadataMtime, GetAudioCache, GetProgress: log warn on PocketBase failures
- EnsureCollections: log info/debug/warn per outcome instead of _ = err
- CountChapterIdx: log warn on failure instead of silently returning 0
- server: log warn when SetAudioCache fails after audio generation
- NewHybridStore: add explicit Ping() before EnsureCollections for fast-fail on bad credentials
novelfire.net chapter-list pages (/chapters?page=N) are server-rendered —
verified via curl. Switch urlClient to NewDirectHTTPClient alongside the
existing chapterClient. Remove BROWSERLESS_URL_STRATEGY env var and clean
up the now-irrelevant WaitFor/GotoOptions fields from both ScrapeChapterList
and ScrapeChapterListPage ContentRequests.
novelfire.net chapter content is server-rendered, so Browserless is not
needed. Add a dedicated chapterClient (always StrategyDirect) to Scraper
and use it in ScrapeChapterText, removing the now-irrelevant WaitFor /
RejectResourceTypes / GotoOptions fields from the ContentRequest.
PocketBase v0.23+ requires 'Bearer <token>' — sending the raw JWT without
the prefix results in 403 'Only superusers can perform this action' on all
collection record endpoints. Fix applied in three places:
- ui/src/lib/server/pocketbase.ts (pbGet, pbPost, pbPatch helpers)
- scraper/internal/storage/pocketbase.go (pbClient.do)
- scripts/pb-init.sh (wget --header in create_collection)
Add scripts/pb-init.sh — an idempotent alpine sh script that authenticates
with the PocketBase admin API and POSTs all required collection schemas
(books, chapters_idx, ranking, progress, audio_cache, app_users) before
any application service starts. 400/422 responses are treated as success so
it is safe to run on every docker compose up.
Wire pb-init into docker-compose.yml:
- pb-init service: alpine:3.19, depends on pocketbase healthy, mounts script
- scraper and ui both depend on pb-init via service_completed_successfully,
guaranteeing collections exist before the first request hits app_users
- e2e fixture: replace single contentClient with directClient (plain HTTP)
for chapter/metadata/ranking + contentClient (Browserless) for urlClient
only — matches production wiring and is significantly faster
- server: add max_chars field to audio request body; truncates stripped text
to N runes before sending to Kokoro (used by e2e for quick TTS tests)
- fix: move RankingItem to scraper package to break novelfire→storage import
cycle; storage.RankingItem is now a type alias for backward compat
- fix: update stale New() call in novelfire integration test (missing args)
- fix: replace removed blob-ranking methods in storage integration test with
current per-item API (UpsertRankingItem/ListRankingItems/RankingLastUpdated)
- justfile: add test-e2e and e2e tasks
- Replace SetRanking/GetRanking/SetRankingPageHTML/GetRankingPageHTML blob methods
with WriteRankingItem/ReadRankingItems/RankingFreshEnough per-item operations
- Add 24h staleness gate in ScrapeRanking to skip re-scraping fresh data
- Add GET /api/ranking endpoint returning []RankingItem sorted by rank
- Remove RankingPageCacher interface and rankingCacheAdapter adapter
- Update integration tests to use new per-item upsert semantics
- Include e2e test suite (scraper/internal/e2e/)
- Replace deprecated /api/admins/auth-with-password with
/api/collections/_superusers/auth-with-password in both the
SvelteKit UI (pocketbase.ts) and Go scraper (pocketbase.go)
- Rename custom users collection to app_users to avoid name clash
with PocketBase's built-in users auth collection
- Fix docker-compose volume mount path pb/pb_data -> pb_data so
persisted data matches the entrypoint --dir flag
- Add PB_ADMIN_EMAIL/PB_ADMIN_PASSWORD env vars to pocketbase service
so the superuser is auto-created on first boot
- scraper/internal/storage/integration_test.go: MinioClient and PocketBaseStore
integration tests covering chapter/audio round-trips, presign URLs, book
metadata, chapter index, ranking, progress, and audio cache CRUD
- scraper/internal/storage/hybrid_integration_test.go: HybridStore end-to-end
tests for metadata, chapters, ranking, progress, presign, and audio cache
- scraper/internal/storage/scrape_integration_test.go: live Browserless + storage
tests that scrape book metadata and first 3 chapters, store them, and verify
the round-trip via HybridStore
- scraper/internal/server/integration_test.go: HTTP server functional tests for
health, scrape status, presign chapter, reading progress, and chapter-text
endpoints against real MinIO + PocketBase backends
- justfile: task runner at repo root with recipes for build, test, lint, UI,
docker-compose, and individual service management
Introduces a logger.ts module emitting slog-compatible JSON lines to stderr.
Replaces silent catch blocks and console.error calls throughout minio.ts,
pocketbase.ts, hooks.server.ts, login, books, browse, and all API routes so
auth/registration failures, MinIO presign errors, and scraper proxy failures
are now visible in container logs.
- /browse calls GET /api/browse on the scraper and renders a novel grid mirroring the
novelfire layout: cover, rank/rating badges, chapter count, genre/sort/status filters,
and pagation controls
- Scrape buttons are shown only to admin users; clicking enqueues the book via /api/scrape
- /api/scrape is an admin-only SvelteKit server route that proxies POST requests to the
Go scraper's /scrape/book or /scrape endpoints; returns 403 for non-admins
- GET /api/browse fetches novelfire.net catalogue page and parses it with golang.org/x/net/html;
returns JSON {novels, page, hasNext} with per-novel slug/title/cover/rank/rating/chapters/url.
Supports page, genre, sort, status query params.
- GET /api/scrape/status returns {"running": bool} for polling job state from the UI
- Add `users` PocketBase collection (username, password_hash, role, created)
- Implement HMAC-SHA256 signed cookie auth in hooks.server.ts; token payload is userId:username:role
- Add User type, getUserByUsername, createUser (scrypt), loginUser (timing-safe) to pocketbase.ts
- Add login/register page with tabbed form UI and server actions
- Add logout route that clears the auth cookie
- Add layout.server.ts auth guard: redirect unauthenticated users to /login
- Extend App.Locals and App.PageData with role field
- Add AUTH_SECRET, POCKETBASE_ADMIN_EMAIL/PASSWORD to .env.example
- Install @types/node for Node crypto/scrypt types
