novelfire.net responds with Content-Encoding: br when the scraper
advertises 'gzip, deflate, br'. The client only handled gzip, so
Brotli-compressed bytes were fed raw into the HTML parser producing
garbage — empty titles, zero chapters, and selector failures.
Added github.com/andybalholm/brotli and wired it into GetContent
alongside the existing gzip path.
Upgrade DirectHTTPClient to send a full Chrome 124 header set
(Sec-Fetch-*, Accept-Encoding with gzip decompression, Referer,
Cache-Control) to reduce bot-detection false positives on WAFs.
Add SCRAPER_PROXY env var to route all outbound scrape requests
through a configurable proxy (residential or otherwise); falls back
to the standard HTTP_PROXY / HTTPS_PROXY env vars.
- Add UserProfileView and UserProfileViewModel for iOS
- Implement user library API endpoint (/api/users/[username]/library)
- Add DELETE /api/progress/[slug] endpoint for removing books from library
- Integrate subscription feed in home API
- Update Xcode project with new profile components
- PocketBase: new user_subscriptions collection (follower_id, followee_id)
- pocketbase.ts: subscribe/unsubscribe/getFollowingIds/getPublicProfile/
getUserPublicLibrary/getUserCurrentlyReading/getSubscriptionFeed helpers
- GET /api/users/[username] — public profile with subscription state
- POST/DELETE /api/users/[username]/subscribe — follow/unfollow
- /users/[username] — public profile page: avatar, stats, follow button,
currently reading grid, full library grid
- CommentsSection: usernames are now links to /users/[username]
- Home page: 'From People You Follow' section powered by subscription feed
iOS BookDetailView: replace paginated inline chapter list with a single
tappable 'Chapters' row (showing reading progress) that opens
BookChaptersSheet — a searchable full-screen sheet with jump-to-current.
ChapterReaderView: hide tab bar in reader, swap back/Aa/ToC button order
(Aa left, ToC right, X rightmost), remove mini-player spacer (tab bar
and player are hidden).
HomeView: remove large HeroContinueCard, promote all continue-reading
items into a single horizontal shelf (Apple Books style) with progress
bar below each cover. NavigationLink now goes directly to the chapter.
Web +page.svelte: replace inline paginated chapter list with a compact
'Chapters' row linking to /books/[slug]/chapters. Admin scrape controls
are now a collapsible row inside the same card.
All four workflows (ci-scraper, ci-ui, release-scraper, release-ui) now pass
build-args to docker/build-push-action. Release workflows use the semver tag
from docker/metadata-action outputs.version; CI workflows use the git SHA.
The /api/comments/[id] delete route was never created; the deleteComment helper
in pocketbase.ts existed but was unreachable. Added DELETE /api/comment/[id]
route handler alongside the existing vote route. Updated CommentsSection.svelte
and iOS APIClient to use /api/comment/{id} for both delete and (already fixed)
vote, keeping all comment-mutation endpoints under the singular /api/comment/
prefix to avoid SvelteKit route conflicts with /api/comments/[slug].
- Go scraper: Version/Commit vars in main.go, injected via -ldflags; Server struct + New() updated; GET /health and new GET /api/version expose them
- UI Dockerfile: ARG BUILD_VERSION/BUILD_COMMIT → ENV PUBLIC_BUILD_VERSION/PUBLIC_BUILD_COMMIT for SvelteKit
- Footer: shows version+short commit when not 'dev' (text-zinc-800, subtle)
- docker-compose: args blocks for scraper and ui build sections pass $GIT_TAG/$GIT_COMMIT
/api/comments/[id] and /api/comments/[slug] were ambiguous dynamic segments at
the same path level, causing a build error. Moved the vote handler to the
singular /api/comment/ prefix and updated all callers (web + iOS).
- Batch-resolve avatar presign URLs server-side in GET /api/comments/[slug];
returns avatarUrls map alongside comments and myVotes
- CommentsSection.svelte: show avatar image or initials fallback (24px top-level,
20px replies) next to each comment/reply username
- iOS CommentsResponse gains avatarUrls field; CommentsViewModel stores and
populates it on load; CommentRow renders AsyncImage with initials fallback
- Also includes: comment replies (1-level nesting), delete, sort (Top/New),
parent_id schema migration, and AvatarCropModal cropperjs fix
Production Docker image has no node_modules at runtime; marked was being
externalized by adapter-node (it is in dependencies), causing ERR_MODULE_NOT_FOUND
when +page.server.ts and +server.ts imported it. Adding it to ssr.noExternal
forces Vite to inline it into the server bundle.
marked({ async: true }) triggers a dynamic internal require inside marked
that vite/rollup treats as external, causing ERR_MODULE_NOT_FOUND at runtime
in the adapter-node Docker image which ships no node_modules.
Switching to the synchronous marked() call makes rollup inline the full
library into the server chunk.
Static import of 'marked' was included in the SSR bundle causing
ERR_MODULE_NOT_FOUND on first server render. The import is only
ever used inside onMount (client-only fallback path), so replace
with a dynamic import() at the call site.
- Add AvatarCropView: fullscreen pan/pinch sheet with circular crop overlay,
outputs 400×400 JPEG at 0.9 quality matching the web crop modal
- ProfileView: picker now shows crop sheet before uploading instead of direct upload
- AuthStore.validateToken: exchange raw MinIO key from /api/auth/me for a
presigned GET URL so avatar renders correctly on cold launch / re-login
- APIClient: add fetchAvatarPresignedURL() calling GET /api/profile/avatar
- Models: add memberwise init to AppUser for avatar URL replacement
- Remove dead structs: ReadingProgress, HomeData, PBList (Models.swift)
- Remove unused APIClient members: fetchRaw, setSessionId, BrowseParams; simplify logout(); drop no-op CodingKeys from BrowseResponse
- Remove absolutePrevChapter/absoluteNextChapter computed props (replaced by prevChapter/nextChapter); replace URLSession cover prefetch with Kingfisher
- Drop scrollOffset state var and dead subtitle block in ChapterRow (BookDetailView)
- Remove never-set chaptersLoading published prop (BookDetailViewModel)
- Add unified ChipButton(.filled/.outlined) to CommonViews replacing three near-identical pill chip types
- Replace FilterChipView+SortChip in LibraryView and FilterChip in BrowseView with ChipButton
- Replace inline KFImage usage in HomeView with AsyncCoverImage
- Fix deprecated .navigationBarHidden(true) → .toolbar(.hidden, for: .navigationBar) in AuthView
- Go scraper: add PresignAvatarUploadURL/PresignAvatarURL/DeleteAvatar to
Store interface, implement on HybridStore+MinioClient, register
GET /api/presign/avatar-upload/{userId} and /api/presign/avatar/{userId}
- SvelteKit: replace direct AWS S3 SDK in minio.ts with presign calls to
the Go scraper; rewrite avatar +server.ts (POST=presign, PATCH=record key)
- iOS: rewrite uploadAvatar() as 3-step presigned PUT flow; refactor
chip components into shared ChipButton in CommonViews.swift
Home:
- Hero card redesign: deeper amber-tinted gradient, taller cover (96×138), inline progress bar + completion % text above CTA
- Continue Reading shelf: replace chapter badge with circular progress arc ring (amber) containing chapter number
- Stats strip: add SF Symbol icons (books, alignleft, bookmark) above each stat value
Reader:
- Strip duplicate chapter-number/date/title prefix that novelfire embeds at top of HTML body
- Bottom chrome: pill-shaped prev/next chapter buttons (amber fill for next, muted for prev), placeholder spacers keep Listen button centered
- Chapter title page: show 'N% through' alongside date label, animated swipe-hint arrow that pulses once on appear
- Progress bar: 2px → 1px for a more refined look
- Outer GeometryReader for adaptive cover sizing (min of width-56 / height*0.42)
- Generating state: cover dims with 0.45 overlay + spinner; seek bar stays visible at opacity 0.3
- Title block: left-aligned with auto-next infinity.circle toggle on far right
- Removed ellipsis menu, year/cache metadata row, showingSpeedMenu state var
- New PlayerSecondaryButton and PlayerChapterSkipButton private helper structs
- Bottom toolbar: AirPlay | Speed | chevron.down | list.bullet | moon (44pt touch targets)
- ignoresSafeArea moved to outermost GeometryReader level
- Reader: list.bullet ToC button in top chrome opens ChaptersListSheet
- Reader: swipe right on title page → prev chapter, swipe left on end page → next chapter
- Reader: scroll mode toggle in settings panel (ReaderSettings.scrollMode); ScrollReaderContent for continuous layout
- Library: segmented All/In Progress/Completed filter
- Library: genre filter chips derived from book.genres, amber fill when active
- Library: completed books show checkmark badge + Finished label
- Library: context-aware empty state messages per filter combination
- Player: prev/next chapter buttons in mini + full player, ±15s skip, sleep timer countdown, Chapter N of M label
- Player: ChaptersListSheet with 100-chapter blocks, jump bar, search filter
- Home/BookDetail/Browse: Apple Books-style redesign, zoom transitions
PROVISIONING_PROFILE accepts the UUID directly and finds the profile correctly.
PROVISIONING_PROFILE_SPECIFIER expects 'TEAM_ID/name' format which caused lookup failures.
Successfully tested locally - archive builds and signs correctly.
The issue was that these settings need to be target-specific in project.yml,
not passed globally via xcodebuild args (which would conflict with SPM dependencies).
Successfully tested locally - archive builds and signs correctly.
Extract currentTime/duration/isPlaying into a separate PlaybackProgress
ObservableObject so the 0.5s time-observer ticks only invalidate the seek
bar and play-pause button subviews, not the entire FullPlayerView or
MiniPlayerView. Menus no longer blink or lose their checkmarks during
playback.
Also fix chapter list selection in FullPlayerView to call audioPlayer.load()
directly instead of relying on skip notifications (which only worked when
ChapterReaderView was open).
Previously, prev/next buttons were relative to the currently playing
chapter and only worked for one navigation step. They relied on
audioPlayer.prevChapter and audioPlayer.nextChapter which were set
when loading audio and never updated.
Now buttons use absolute chapter navigation:
- Previous: always navigate to (current chapter - 1)
- Next: always navigate to (current chapter + 1)
- Bounded by chapter 1 and max chapter number
- Works continuously without stopping after one navigation
Added AudioPlayerService.absolutePrevChapter and .absoluteNextChapter
computed properties that calculate the prev/next chapter numbers based
on the current chapter and total chapters list.
Updated both mini player and full player to use absolute navigation.
Change seek gesture from .gesture to .highPriorityGesture to ensure
horizontal press-and-drag seeking takes precedence over the vertical
swipe gesture. This fixes unreliable seeking when dragging on the
mini player progress bar.
- Moved audio start button from top-right toolbar to floating bottom-right position
- Only shows floating button when audio player is not active
- Button styled as amber pill with 'Listen' text and play icon
- More accessible placement for quick audio playback start
- Positioned 20pt from bottom and right edges with shadow
- Created 1024x1024 app icon with amber/orange gradient background
- Added white open book icon in center with pages and spine
- Gradient matches app's amber accent color theme
- Clean, minimal, modern design suitable for iOS
- Added previous chapter button (matching next button style)
- Made progress bar interactive - now occupies full background of mini player
- Horizontal drag/tap on progress bar seeks audio (shows amber overlay while seeking)
- Progress bar now uses full pill shape as background instead of small bottom border
- Tightened button spacing (12pt instead of 16pt) to fit prev/next buttons
- Changed vertical drag to simultaneousGesture to not conflict with horizontal seek
- Buttons slightly smaller (40pt) to accommodate prev button in layout
- Reduced bottom padding from 24pt to 8pt on bottom toolbar
- Reduced spacer above toolbar from 24pt to 16pt
- Added .ignoresSafeArea(edges: .bottom) to VStack to extend to screen bottom
- Player now uses full screen height without excess bottom padding
- Added custom init to ChaptersListSheet that sets scrollPosition state before view appears
- Removed onAppear scroll position update (now set in init)
- List renders directly at current chapter position without any scroll animation
- Better performance for books with 1000+ chapters (no initial render + animate delay)
- Changed progress bar from RoundedRectangle to Capsule for rounded ends
- Added horizontal padding so indicator wraps nicely around bottom corners
- Progress bar now feels like an integrated part of the pill-shaped mini player
- Full player prev/next buttons now dismiss the player before navigating (allows user to see chapter change)
- Added amber loading indicator on next chapter button when prefetching audio (both mini and full player)
- Indicator appears as small ProgressView in bottom-right corner of forward button
- Fix chapter-based sleep timer not resetting when auto-advancing to next chapter (sleepTimerStartChapter now updates in handlePlaybackFinished)
- Fix AirPlay button being too large compared to other toolbar icons (constrain to 22×22)
Replace ScrollViewReader with scrollPosition to instantly show the current chapter without visible scrolling. This scales better for books with 1000+ chapters.
AirPlay:
- Replaced placeholder AirPlay button with AVRoutePickerView
- Users can now cast audio to AirPlay devices
- Icon changes color when AirPlay is active
Sleep Timer:
- Added sleep timer sheet with multiple options
- Chapter-based: 1, 2, 3, or 4 chapters
- Time-based: 20 mins, 40 mins, 1 hour, 2 hours
- Moon icon fills when timer is active and turns amber
- Timer stops playback when target is reached
- Chapter-based timer tracks chapters played from start
- Time-based timer uses async task with proper cancellation
- Timer is cancelled when playback stops manually
- SleepTimerOption enum supports both timer types
- List bullet button now opens a chapters list sheet
- ChaptersListSheet displays all chapters with scroll view
- Current chapter highlighted with amber badge and background
- Auto-scrolls to current chapter when opened
- Chapter selection uses skip notifications to navigate
- Supports both .medium and .large presentation sizes
- Shows "Now Playing" label on current chapter
- Clean list design with chapter numbers and titles
- Fully pill-shaped design (cornerRadius 40) with dark glassmorphic background
- Larger cover thumbnail (56×56 with 12pt corner radius)
- Improved layout: cover + chapter/book titles + play/pause + next chapter skip
- Progress indicator moved to bottom 3pt amber border
- Removed dismiss X button - swipe down to dismiss instead
- Added swipe gestures: up to expand (0.4x resistance), down to dismiss (0.8x resistance)
- Dismiss animation with fade and scale effects
- Skip forward button navigates to next chapter (disabled when unavailable)
- Updated mini player height to 92pt in AppLayout
- Add safeAreaInset on ChapterReaderView ScrollView so Prev/Next buttons
clear the mini player bar when audio is active
- Fix mini player height reservation (66pt constant via AppLayout)
- Add AsyncCoverImage(isBackground:) to suppress rounded-rect placeholder
showing through blurred hero background (empty square bug)
- Reduce chapter page size 100→50 on iOS
- Fix ChapterRow date/chevron layout: minLength spacer + fixedSize date
- Add String.strippingTrailingDate() extension; apply in MiniPlayer,
FullPlayer, and ChapterReader header to clean up stale concatenated titles
- Make Prev/Next buttons equal-width (maxWidth: .infinity) for easier tapping
- Replace AudioGenerateResponse with AudioTriggerResponse to handle both 200 (cached) and 202 (async job) responses
- Add pollAudioStatus() in APIClient that polls /api/audio/status every 2s until done or failed
- Update generateAudio() and prefetchNext() in AudioPlayerService to poll instead of expecting a synchronous URL response
- Bump chapter list font sizes: number xs→sm, title sm→base, date/badge xs→sm, row padding py-2→py-2.5
New page mirrors the scrape tasks pattern: loads audio_jobs from
PocketBase via a new listAudioJobs() helper, shows a filterable table
(slug, chapter, voice, status, started, duration, error), and live-polls
every 3s while any job is pending or generating. Also fixes the
/admin/audio nav active-state check (was startsWith, now exact match)
to prevent it from matching /admin/audio-jobs.
- AudioPlayerService: replace loading/generating states with a single
.generating state; presign fast path before triggering TTS; fix URL
resolution for relative paths; cache cover artwork to avoid re-downloads;
fix duration KVO race (durationObserver); use toleranceBefore/After:zero
for accurate seeking; prefetch next chapter unconditionally (not just when
autoNext is on); handle auto-next internally on playback finish
- APIClient: fix URL construction (appendingPathComponent encodes slashes);
add verbose debug logging for all requests/responses/decoding errors;
fix sessions() to unwrap {sessions:[]} envelope; fix BrowseResponse
CodingKey hasNext → camelCase
- AudioGenerateResponse: update to synchronous {url, filename} shape
- Models: remove redundant AudioStatus enum; remove CodingKeys from
UserSettings (server now sends camelCase)
- Views: fix alert bindings (.constant → proper two-way Binding); add
error+retry UI to BrowseView; add pull-to-refresh to browse list;
fix ChapterReaderView to show error state and use dynamic WKWebView
height; fix HomeView HStack alignment; only treat audio as current
if the player isActive; suppress CancellationError from error UI
Replace blocking POST /api/audio with a non-blocking 202 flow: the Go
handler immediately enqueues a job in a new `audio_jobs` PocketBase
collection and returns {job_id, status}. A background goroutine runs
the actual Kokoro TTS work and updates job status (pending → generating
→ done/failed). A new GET /api/audio/status/{slug}/{n} endpoint lets
clients poll progress. The SvelteKit proxy and AudioPlayer.svelte are
updated to POST, then poll the status route every 2s until done.
- justfile: extract ios_scheme/ios_sim/ios_spm/runner_temp variables; add
ios-resolve (SPM dep cache), ios-archive, ios-export recipes; pipe xcodebuild
through xcpretty with raw fallback in ios-build/ios-test
- ios.yaml: replace raw xcodebuild calls with just ios-build / just ios-test;
install just via brew; uncommented archive job stub now calls just ios-archive
&& just ios-export; add justfile to path trigger
- ci-scraper.yaml: install just via extractions/setup-just@v2; use just lint /
just test; fix upload-artifact v3 -> v4; add justfile to path trigger
- ci-ui.yaml: install just; use just ui-install / just ui-check / just ui-build;
add justfile to path trigger
- Add user_sessions PocketBase collection (user_id, session_id, user_agent, ip, created/last_seen)
- Extend auth token format to include a per-login authSessionId (4th segment)
- Hook validates authSessionId against DB on each request; revoked sessions are cleared immediately
- Login/register create a session record capturing user-agent and IP
- Profile page shows all active sessions with current session highlighted; per-session End/Sign out buttons
- GET /api/sessions and DELETE /api/sessions/[id] endpoints for client-side revocation
- Backward compatible: legacy 3-segment tokens pass through without DB check
The MinIO cache key only encoded page number, so all filter combinations
hit the same cache entry and returned unfiltered results. Introduce
BrowseFilteredHTMLKey() that encodes sort/genre/status into the key
(e.g. novelfire.net/html/new-isekai-completed/page-1.html); falls back
to the original page-only key for default filters to preserve existing
cached pages.
- Store cover and chapters array in audioStore (populated by AudioPlayer on startPlayback)
- Page server returns full chapters list on normal path; empty array on preview
- Mini-bar: replace arrow link with book cover thumbnail (fallback book icon)
- Mini-bar track info area is now a button that toggles a chapter list drawer
- Chapter drawer slides up above the mini-bar, lists all chapters with current one highlighted
When a book is not in the local DB and the preview endpoint is hit,
metadata and the chapter index are now saved to PocketBase in the
background. Subsequent visits load from the local store instead of
scraping live. Chapter text is not fetched until an explicit scrape job
is triggered.
- Production deploy: POST /api/trpc/compose.redeploy on push to main
- Preview deploy: POST /api/trpc/compose.isolatedDeployment on feature branches
- Preview cleanup: compose.search + compose.delete on PR close
- Uses x-api-key auth and tRPC v10 batch JSON body format
Adds GET /api/search Go endpoint that queries PocketBase books by title/author
substring and fetches novelfire.net /search results via parseBrowsePage(),
merging results with local-first de-duplication. The browse page gains a search
input that navigates to ?q=; results show local vs remote counts and hide
pagination/filter controls while in search mode.
Adds FromChapter/ToChapter fields to orchestrator.Config and skips out-of-range
chapters in processBook. Exposes POST /scrape/book/range Go endpoint and a
matching UI proxy at /api/scrape/range. The book detail page now shows admins
a range input (from/to chapter) and a per-chapter 'scrape from here up' button.
Adds GET /api/book-preview/{slug} and GET /api/chapter-text-preview/{slug}/{n}
Go endpoints that scrape live from novelfire.net without persisting to
PocketBase or MinIO. The UI book page falls back to the preview endpoint when
a book is not found in PocketBase, showing a 'not in library' badge and the
scraped chapter list. Chapter pages handle ?preview=1 to fetch and render
chapter text live, skipping MinIO and suppressing the audio player.
- Remove SingleFile CLI and Node.js from Dockerfile; runtime base reverted to alpine:3.21
- Replace triggerBrowseSnapshot with triggerDirectScrape: plain Go HTTP GET to novelfire.net
(page is server-rendered, no browser needed)
- Fix Accept-Encoding bug: stop setting header manually so Go transport auto-decompresses gzip
- Add in-memory browse cache fallback (browseMemCache) for when MinIO and upstream both fail
- Add warmBrowseCache() startup goroutine
- Call triggerDirectScrape on MinIO cache hits too, so PocketBase ranking records are
repopulated after a schema reset or fresh deploy without waiting for a cache miss
- Fix grid view cards: wrap in <a> instead of <div> so clicking navigates to book detail
- Remove SINGLEFILE_PATH and BROWSERLESS_URL env vars from Dockerfile
The previous implementation sent a single request with perPage=500 and
silently dropped any records beyond that. Books with 900+ chapters were
truncated to 500 in the chapter list and reader prev/next navigation.
Now loops through all pages until totalItems is exhausted.
- Speed is no longer part of Kokoro generation, in-memory cache keys, or
MinIO object keys — audio is always generated at 1.0 and playback speed
is applied client-side via audioEl.playbackRate
- presignAudio() now calls rewriteHost() so chapter audio URLs use the
public MinIO endpoint (same as presignVoiceSample already did)
- docker-compose.yml: rename MINIO_PUBLIC_ENDPOINT → PUBLIC_MINIO_PUBLIC_URL
for the ui service so SvelteKit's $env/dynamic/public picks it up
Add warmVoiceSamples() goroutine launched from ListenAndServe.
It waits up to 30s for Kokoro to become reachable, then generates
missing voice sample clips for all available voices and uploads them
to MinIO (_voice-samples/{voice}.mp3). Already-existing samples are
skipped, so the operation is idempotent on restarts.
- Fix speed/voice bug: AudioPlayer no longer accepts speed/voice as props;
startPlayback() reads audioStore.voice/speed directly instead of overwriting them
- Add GET /api/voices endpoint (Go) proxying Kokoro, cached in-memory
- Add POST /api/audio/voice-samples endpoint (Go) to pre-generate sample clips
for all voices and store them in MinIO under _voice-samples/{voice}.mp3
- Add GET /api/presign/voice-sample/{voice} endpoint (Go)
- Add SvelteKit proxy routes: /api/voices, /api/presign/voice-sample, /api/audio/voice-samples
- Add presignVoiceSample() helper in minio.ts with proper host rewrite
- Pass book.cover through +page.server.ts -> +page.svelte -> AudioPlayer
- Set navigator.mediaSession.metadata on playback start so cover art,
book title, and chapter title appear on phone lock screen / notification
Admins see a Rescrape button next to the chapter list header.
Clicking it POSTs to /api/scrape with the book's source_url,
then shows an inline status banner (queued / busy / error).
isAdmin is exposed from the server load; no new routes needed.
The boolean flag was set by onended before goto() resolved, causing the
still-mounted outgoing chapter's AudioPlayer $effect to fire and call
startPlayback() for the wrong (old) chapter — restarting it from scratch.
Replace with autoStartChapter (number | null): the AudioPlayer only acts
when its own chapter prop === autoStartChapter, so the outgoing component
never matches and the incoming one fires exactly once on mount.
The stale-prefetch reset effect compared prefetchedFor against nextChapter
only, so when landing on chapter N+1 (prefetchedFor=N+1, nextChapter=N+2)
it would destroy the pre-fetched URL before startPlayback() could use it,
breaking every auto-next transition after the first.
Fix: also allow prefetchedFor === chapter (current page) so the URL
survives long enough to be consumed, then only wipe truly foreign values.
When autoNext is on, kick off prefetchNext() as soon as the current
chapter begins playing (all three success paths in startPlayback).
This gives the full chapter duration as lead time for Kokoro TTS
instead of only the last 10%, making auto-next transitions seamless.
The existing 90%-mark $effect is kept as a fallback for cases where
autoNext is toggled on mid-playback after startPlayback has returned.
- Fix audio presign 404: MinIO upload is now synchronous before response is sent,
eliminating the race where presign was called before the file landed in MinIO
- Replace all Browserless usage with direct HTTP client across catalogue, metadata,
ranking, and browse — novelfire.net pages are server-rendered and don't need a
headless browser; direct is faster and more reliable
- Harden handleBrowse with 3-attempt retry loop, proper backoff, and full
browser-like headers to reduce 502s from novelfire.net bot detection
- Remove Browserless env vars (BROWSERLESS_URL/TOKEN/STRATEGY) from main.go;
add SCRAPER_TIMEOUT as a single timeout knob
- Clean up now-dead rejectResourceTypes var and Browserless-specific WaitFor/
RejectResourceTypes/GotoOptions fields from scraper calls
- Add NextStatus type and prefetch state (nextStatus, nextAudioUrl, nextProgress, nextChapterPrefetched) to AudioStore
- AudioPlayer triggers prefetchNext() when currentTime/duration >= 0.9, autoNext is on, and a next chapter exists
- startPlayback() uses the pre-fetched URL if available (skipping presign round-trip and generation wait)
- Auto-next button in layout shows a pulsing dot while prefetching and a green dot when ready
- AudioPlayer shows inline prefetch progress and ready state below the controls
- Fix indentation regression in layout onended handler
- ScrapeCatalogue: use li.novel-item (was div), extract href from outer <a> and title from h4.novel-title (was h3), detect next page via rel=next (was class=next)
- handleBrowse/triggerBrowseSnapshot: treat zero-byte MinIO cache entries as misses, skip storing empty SingleFile output
- runAsync: after a successful full-catalogue run, call ScrapeRanking and upsert each result into PocketBase ranking collection
- Clear autoStartPending if goto() fails to avoid spurious auto-starts on future navigations
- Clear audioStore.nextChapter on AudioPlayer unmount so a stale chapter can't trigger navigation after leaving a chapter page
- Remove redundant nextChapter write in startPlayback() — the already keeps it in sync
- Replace double inline player controls with compact 'now playing' indicator when mini-bar is active
- Add user_settings PocketBase collection (auto_next, voice, speed) and audio_time field on progress
- Add getSettings/saveSettings/setAudioTime/getAudioTime server functions
- Add GET/PUT /api/settings and GET/PATCH /api/progress/audio-time API routes
- Load settings server-side in layout and apply to audioStore on mount
- Debounced 800ms effect persists settings changes to DB
- Save audio currentTime on pause/end; restore position when replaying a chapter
When autoNext is enabled, audio automatically advances to the next chapter
when a track ends — navigating the page and starting the new chapter's audio.
- audioStore: add autoNext (toggle flag), nextChapter (written by AudioPlayer),
and autoStartPending (set before goto, cleared after auto-start fires)
- layout: update onended to call goto() and set autoStartPending when autoNext
is on and nextChapter is available; add auto-next toggle button to mini-player
bar (double-chevron icon, amber when active)
- AudioPlayer: accept nextChapter prop; write it to audioStore via $effect;
auto-start playback when autoStartPending is set on component mount; show
inline 'Auto' toggle button in ready controls when a next chapter exists
- Chapter page: pass data.next as nextChapter prop to AudioPlayer
Two bugs caused the start/stop loop:
1. The <audio> element was wrapped in {#if audioStore.audioUrl}, so whenever
any reactive state changed (e.g. currentTime ticking), Svelte could destroy
and recreate the element, firing onpause and then the URL effect restarting
playback.
2. Comparing audioEl.src !== url is unreliable — browsers normalise the src
property to a full absolute URL, causing false mismatches every tick.
Fix: make <audio> always present in the DOM (display:none), and track the
loaded URL in a plain local variable (loadedUrl) instead of reading audioEl.src.
- Add audio.svelte.ts: module singleton AudioStore (Svelte 5 runes) with
slug/chapter/title metadata, status, progress, playback state, and
toggleRequest/seekRequest signals for layout<->audio element communication
- Rewrite AudioPlayer.svelte as a store controller: no <audio> element owned;
drives audioStore for presign->generate->play flow; shows inline controls
when current chapter is active, 'Now playing / Load this chapter' banner
when a different chapter is playing
- Update +layout.svelte: single persistent <audio> outside {#key} block so
it never unmounts on navigation; effects to load URL, sync speed, handle
toggle/seek requests; fixed bottom mini-player bar with seek, skip 15s/30s,
speed cycle, go-to-chapter link, dismiss; pb-24 padding when active
- Pass chapterTitle and bookTitle from chapter page to AudioPlayer
handlePresignAudio now checks AudioExists before presigning; previously it
would generate a valid-looking signed URL for a non-existent object, causing
the browser to get 404 from MinIO directly.
- Add AudioExists to Store interface and HybridStore
- Guard handlePresignAudio with AudioExists check, return 404 if missing
- Propagate 404 through presignAudio() in minio.ts (typed error.status=404)
- SvelteKit presign route forwards 404 to the browser instead of 500
- Single 'Play narration' button: checks presign first, plays immediately if audio exists, otherwise triggers generation
- During generation shows an animated pseudo progress bar: slow start (4%/s) → accelerates (12%/s at 30%) → slows near 80% (4%/s) → crawls to 99% (0.3%/s)
- When generation completes, bar jumps to 100% then transitions to audio player
- Auto-plays audio after generation completes
On every scraper startup, EnsureMigrations fetches each collection schema
and PATCHes in any missing fields. This repairs the progress collection on
cloud deploys where pb-init ran before user_id was added to the schema.
No-ops when the field already exists.
The previous ensure_fields helper used python3 which is not available in
alpine:3.19, causing exit 127 in the cloud init container.
Replaced with ensure_field (per-field, no python/jq) that uses only
busybox sh, wget, sed, and awk. Also fixed the HTTP status grep pattern
in create_collection and ensure_field to match only the response header
line (^ HTTP/) instead of the wget error line, eliminating the spurious
'unexpected status server' warnings.
The progress collection was created before user_id was added to the schema,
causing all user-keyed progress queries to return 400. Add an ensure_fields
helper that idempotently patches existing collections with missing fields,
and run it for progress.user_id on every init.
Replace the single-button placeholder with a full homepage:
- Stats bar: total books, total chapters, books in progress
- Continue Reading grid (up to 6): links directly to last chapter read
- Recently Updated grid (up to 6): recent books not already in progress
- Empty state with Discover Novels CTA when library is empty
New pocketbase.ts helpers: recentlyAddedBooks(), getHomeStats(),
listN(), countCollection().
The pre-compiled single-file binary requires glibc symbols (__res_init)
that Alpine's gcompat shim does not provide, causing exit status 127.
Switch runtime base to node:22-alpine and install single-file-cli via npm.
Also add .dockerignore to exclude bin/ and static/ from build context.
- New MinIO bucket 'libnovel-browse' (MINIO_BUCKET_BROWSE env) for storing
self-contained HTML snapshots of novelfire browse pages
- Store interface gains SaveBrowsePage / GetBrowsePage / BrowsePageKey methods
- handleBrowse is now cache-first: serves from MinIO snapshot when available,
then fires a background triggerBrowseSnapshot goroutine to populate cache
on live-fetch (de-duplicated, 90s timeout)
- New 'save-browse' CLI subcommand to bulk-capture pages via SingleFile CLI
- Dockerfile: downloads pinned single-file-x86_64-linux binary (v2.0.83),
adds gcompat + libstdc++ to Alpine runtime for glibc compatibility
- docker-compose: adds libnovel-browse bucket init and SINGLEFILE_PATH env
- .gitignore: exclude scraper/scraper build artifact
The /ranking route is removed. /browse becomes the Discover page with:
- Sort=Ranking option: fetches from /api/ranking (richer metadata — author,
genres, source_url) and renders the full pre-computed ranked list
- Sort=Popular/New/Updated: fetches from /api/browse as before, paginated
- Grid / List view toggle: grid is the default for browse; list auto-selects
for ranking and gives the information-dense ranked layout (was /ranking)
- Admin Refresh catalogue button (moved from /ranking) and per-novel Scrape
button work in both views
- Genre and status filters are disabled (visually dimmed) when sort=rank
since the ranking endpoint does not support per-page filtering
- Nav: 'Browse' renamed to 'Discover', 'Ranking' link removed
Add optional user_id field to the progress collection. When a user is
authenticated, progress is keyed by user_id instead of session_id, making
it portable across devices and browsers. Anonymous reading still works via
session_id as before.
On login or registration, any progress accumulated under the anonymous
session is merged into the user's account (most-recent timestamp wins),
so chapters read before logging in are not lost.
- scripts/pb-init.sh: add user_id field to progress collection schema
- scraper/internal/storage/pocketbase.go: add user_id to EnsureCollections
- ui/src/lib/server/pocketbase.ts: Progress interface gains user_id;
getProgress/allProgress/setProgress accept optional userId and query/write
by user_id when present; add mergeSessionProgress() helper
- ui/src/routes/login/+page.server.ts: call mergeSessionProgress after
successful login and registration (fire-and-forget, non-fatal)
- ui/src/routes/api/progress/+server.ts: pass locals.user?.id to setProgress
- ui/src/routes/books/+page.server.ts: pass locals.user?.id to allProgress
- ui/src/routes/books/[slug]/+page.server.ts: pass locals.user?.id to getProgress
Add a second MinIO client (pub) initialized with MINIO_PUBLIC_ENDPOINT so
presigned audio URLs are signed against the public hostname from the start,
rather than signed internally and then rewritten. This avoids AWS4 signature
mismatch (SignatureDoesNotMatch 403) that occurred when the signed host was
substituted after signing.
- storage/minio.go: add PublicEndpoint/PublicUseSSL to MinioConfig; add pub
client field; NewMinioClient creates pub client when public endpoint differs;
PresignAudio uses pub, PresignChapter keeps internal client
- cmd/scraper/main.go: wire MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL env vars
- docker-compose.yml: expose MINIO_PUBLIC_ENDPOINT and MINIO_PUBLIC_USE_SSL to scraper service
- ui/src/lib/server/minio.ts: remove rewriteHost() call from presignAudio
The SvelteKit server fetches chapter markdown server-side using the presigned
URL. rewriteHost() was replacing the internal minio:9000 host with the public
PUBLIC_MINIO_PUBLIC_URL (localhost:9000), which is unreachable from inside
Docker, causing MinIO 403/connection errors.
presignChapter now takes an optional rewrite=false parameter — the server-side
load function gets the raw internal URL, while any future browser-facing use
can pass rewrite=true to get the public-facing URL.
- ui/Dockerfile: copy package-lock.json and run npm ci --omit=dev in the
runtime stage so marked (and other runtime deps) are available to
adapter-node at startup — fixes ERR_MODULE_NOT_FOUND for 'marked'
- storage: add ReindexChapters to Store interface and HybridStore — walks
MinIO objects for a slug, reads chapter titles, upserts chapters_idx
- server: add POST /api/reindex/{slug} to rebuild chapters_idx from MinIO
Replace selected={data.x === opt.value} on individual <option> elements
with value={data.x} on the <select> — the idiomatic Svelte approach that
ensures correct hydration and form submission of the current filter values.
Adds /ranking route showing the cached ranking list with cover, title,
author, status, and genres. Admin users get a Refresh button that
triggers a full catalogue scrape. Adds Ranking to the nav.
runAsync creates a scraping_tasks record on job start, flushes progress
counters via OnProgress, and finalizes status (done/failed/cancelled) on
completion. Adds GET /api/scrape/tasks to list all historical jobs.
Also fixes relative cover URLs in parseBrowsePage.
Adds atomic counters for books_found, chapters_scraped, chapters_skipped,
and errors. The new OnProgress callback fires after each counter update
and once more at the end of Run, giving callers a live progress feed.
Adds a scraping_tasks PocketBase collection with fields for kind, status,
progress counters, timestamps, and error info. Exposes CreateScrapeTask,
UpdateScrapeTask, and ListScrapeTasks on the Store interface with
implementations in HybridStore and PocketBaseStore.
- Inject *slog.Logger into HybridStore, PocketBaseStore, and pbClient
- Fix credential defaults in main.go (changeme123 / admin) to match docker-compose
- listOne/listAll/upsert/deleteWhere now return errors on non-2xx HTTP status
- WriteChapter: log warn instead of discarding UpsertChapterIdx error
- MetadataMtime, GetAudioCache, GetProgress: log warn on PocketBase failures
- EnsureCollections: log info/debug/warn per outcome instead of _ = err
- CountChapterIdx: log warn on failure instead of silently returning 0
- server: log warn when SetAudioCache fails after audio generation
- NewHybridStore: add explicit Ping() before EnsureCollections for fast-fail on bad credentials
novelfire.net chapter-list pages (/chapters?page=N) are server-rendered —
verified via curl. Switch urlClient to NewDirectHTTPClient alongside the
existing chapterClient. Remove BROWSERLESS_URL_STRATEGY env var and clean
up the now-irrelevant WaitFor/GotoOptions fields from both ScrapeChapterList
and ScrapeChapterListPage ContentRequests.
novelfire.net chapter content is server-rendered, so Browserless is not
needed. Add a dedicated chapterClient (always StrategyDirect) to Scraper
and use it in ScrapeChapterText, removing the now-irrelevant WaitFor /
RejectResourceTypes / GotoOptions fields from the ContentRequest.
PocketBase v0.23+ requires 'Bearer <token>' — sending the raw JWT without
the prefix results in 403 'Only superusers can perform this action' on all
collection record endpoints. Fix applied in three places:
- ui/src/lib/server/pocketbase.ts (pbGet, pbPost, pbPatch helpers)
- scraper/internal/storage/pocketbase.go (pbClient.do)
- scripts/pb-init.sh (wget --header in create_collection)
Add scripts/pb-init.sh — an idempotent alpine sh script that authenticates
with the PocketBase admin API and POSTs all required collection schemas
(books, chapters_idx, ranking, progress, audio_cache, app_users) before
any application service starts. 400/422 responses are treated as success so
it is safe to run on every docker compose up.
Wire pb-init into docker-compose.yml:
- pb-init service: alpine:3.19, depends on pocketbase healthy, mounts script
- scraper and ui both depend on pb-init via service_completed_successfully,
guaranteeing collections exist before the first request hits app_users
- e2e fixture: replace single contentClient with directClient (plain HTTP)
for chapter/metadata/ranking + contentClient (Browserless) for urlClient
only — matches production wiring and is significantly faster
- server: add max_chars field to audio request body; truncates stripped text
to N runes before sending to Kokoro (used by e2e for quick TTS tests)
- fix: move RankingItem to scraper package to break novelfire→storage import
cycle; storage.RankingItem is now a type alias for backward compat
- fix: update stale New() call in novelfire integration test (missing args)
- fix: replace removed blob-ranking methods in storage integration test with
current per-item API (UpsertRankingItem/ListRankingItems/RankingLastUpdated)
- justfile: add test-e2e and e2e tasks
- Replace SetRanking/GetRanking/SetRankingPageHTML/GetRankingPageHTML blob methods
with WriteRankingItem/ReadRankingItems/RankingFreshEnough per-item operations
- Add 24h staleness gate in ScrapeRanking to skip re-scraping fresh data
- Add GET /api/ranking endpoint returning []RankingItem sorted by rank
- Remove RankingPageCacher interface and rankingCacheAdapter adapter
- Update integration tests to use new per-item upsert semantics
- Include e2e test suite (scraper/internal/e2e/)
- Replace deprecated /api/admins/auth-with-password with
/api/collections/_superusers/auth-with-password in both the
SvelteKit UI (pocketbase.ts) and Go scraper (pocketbase.go)
- Rename custom users collection to app_users to avoid name clash
with PocketBase's built-in users auth collection
- Fix docker-compose volume mount path pb/pb_data -> pb_data so
persisted data matches the entrypoint --dir flag
- Add PB_ADMIN_EMAIL/PB_ADMIN_PASSWORD env vars to pocketbase service
so the superuser is auto-created on first boot
- scraper/internal/storage/integration_test.go: MinioClient and PocketBaseStore
integration tests covering chapter/audio round-trips, presign URLs, book
metadata, chapter index, ranking, progress, and audio cache CRUD
- scraper/internal/storage/hybrid_integration_test.go: HybridStore end-to-end
tests for metadata, chapters, ranking, progress, presign, and audio cache
- scraper/internal/storage/scrape_integration_test.go: live Browserless + storage
tests that scrape book metadata and first 3 chapters, store them, and verify
the round-trip via HybridStore
- scraper/internal/server/integration_test.go: HTTP server functional tests for
health, scrape status, presign chapter, reading progress, and chapter-text
endpoints against real MinIO + PocketBase backends
- justfile: task runner at repo root with recipes for build, test, lint, UI,
docker-compose, and individual service management
Introduces a logger.ts module emitting slog-compatible JSON lines to stderr.
Replaces silent catch blocks and console.error calls throughout minio.ts,
pocketbase.ts, hooks.server.ts, login, books, browse, and all API routes so
auth/registration failures, MinIO presign errors, and scraper proxy failures
are now visible in container logs.
- /browse calls GET /api/browse on the scraper and renders a novel grid mirroring the
novelfire layout: cover, rank/rating badges, chapter count, genre/sort/status filters,
and pagation controls
- Scrape buttons are shown only to admin users; clicking enqueues the book via /api/scrape
- /api/scrape is an admin-only SvelteKit server route that proxies POST requests to the
Go scraper's /scrape/book or /scrape endpoints; returns 403 for non-admins
- GET /api/browse fetches novelfire.net catalogue page and parses it with golang.org/x/net/html;
returns JSON {novels, page, hasNext} with per-novel slug/title/cover/rank/rating/chapters/url.
Supports page, genre, sort, status query params.
- GET /api/scrape/status returns {"running": bool} for polling job state from the UI
- Add `users` PocketBase collection (username, password_hash, role, created)
- Implement HMAC-SHA256 signed cookie auth in hooks.server.ts; token payload is userId:username:role
- Add User type, getUserByUsername, createUser (scrypt), loginUser (timing-safe) to pocketbase.ts
- Add login/register page with tabbed form UI and server actions
- Add logout route that clears the auth cookie
- Add layout.server.ts auth guard: redirect unauthenticated users to /login
- Extend App.Locals and App.PageData with role field
- Add AUTH_SECRET, POCKETBASE_ADMIN_EMAIL/PASSWORD to .env.example
- Install @types/node for Node crypto/scrypt types
- Drop kokoro service (deployed separately); update default KOKORO_URL to kokoro.kalekber.cc
- Expose host ports via env vars (MINIO_PORT, MINIO_CONSOLE_PORT, POCKETBASE_PORT, BROWSERLESS_PORT, SCRAPER_PORT) for preview deployments
