Building Audiobooks: a Mac app that syncs an audiobook with its Kindle ebook, on-device

I read on a Kindle. I also listen to audiobooks while I cook, clean or walk around the apartment. The problem is the moment I switch: I put the Kindle down on page 214, press play on the audiobook, and have no idea where in the next six hours that page is. Then I come back to the Kindle after an hour of listening and have to hunt for the paragraph the narrator stopped on.

Commercial services solve this by owning both the ebook and the audio. My books do not live in one ecosystem: the audio is a folder of files on my Mac, the ebook is an EPUB on my Kindle. So I built the player I wanted, over a couple of days, with Claude Code as the implementation partner.

In short:

• Audiobooks is a native macOS audiobook player (Swift and SwiftUI) that plays MP3, M4A and M4B files straight from a folder.
• It syncs the audiobook with its EPUB on-device: Apple's SpeechAnalyzer transcribes the narration, and four-word phrases that occur only once in the ebook anchor each second of audio to a word of text.
• A Kindle handoff panel gives a phrase that appears once in the whole book. Typing it into Kindle search lands on the exact sentence, and it works in reverse too.
• A six-hour book synced in under three minutes with more than 43,000 anchors. No cloud, no accounts, no Audible.
• Claude Code wrote most of the app and also produced the demo video below: a scripted demo mode, window recording, FFmpeg audio sync, music and a vertical cut.

Here is the result, in under a minute:

The video shows the library as a rail of covers, Read mode highlighting each sentence as it is narrated, the Kindle handoff panel, and focus mode. Captions are in Spanish. Audio: Marianela by Benito Pérez Galdós, read by Maritza Mateo for LibriVox (public domain). Music built from GarageBand Apple Loops.

The idea: an audiobook player that knows the ebook

The first version was a plain native player. SwiftUI, no web views. It plays every .mp3, .m4a or .m4b under ~/Documents/Audiobooks straight from disk, reads the chapter markers embedded in the files, and remembers where I left off in each book. Media keys, AirPods and Control Center all work, because it uses the system's Now Playing APIs instead of reinventing them.

That part is a solved problem. The interesting part was teaching the app where the audio sits in the ebook.

How do you sync an audiobook with an ebook you already own?

There is no API to read the current page from a Kindle, and no shared timeline between an audiobook file and an EPUB. Amazon's Whispersync for Voice gets around this by selling you both halves as matched editions. With files from anywhere, the app has to build that timeline itself:

1. Transcribe the audiobook on-device. Apple's new SpeechAnalyzer and SpeechTranscriber (macOS 26) run fully offline and give every recognized word a timestamp. The transcript is cached, so this happens once per book.
2. Find words that can only be in one place. The app indexes every four-word sequence in the EPUB and keeps only the ones that occur exactly once in the whole book.
3. Match and keep the consistent path. Every unique sequence the narrator says becomes a candidate anchor (this word in the ebook was spoken at this second). Speech recognition makes mistakes and some phrases repeat across chapters, so the app keeps the longest chain of matches that moves forward in both the audio and the text, then drops anchors that jump away from their neighbors.

The result is a dense map between seconds and words. For the public-domain book in the demo, a six-hour recording synced in under three minutes on my Mac with more than 43,000 anchors, roughly two per second of audio. Once that exists, everything else is a lookup. The chapter list switches to the ebook's real chapters, and the app can answer "which sentence is the narrator reading right now".

The app does not need to be told which ebook goes with which audiobook. It tries every EPUB in the library and keeps the one that lines up. A real match aligns a large share of the transcript, and the wrong book barely aligns at all. The transcription language comes from the EPUB's own metadata, which is how the Spanish book in the demo was transcribed in Spanish without any setting.

Transcribing a whole audiobook with SpeechAnalyzer

SpeechAnalyzer is new in macOS 26 and there are not many examples of feeding it a long file, so here is the shape of it. The transcriber asks for audio time ranges, AVAssetReader decodes the audiobook into PCM buffers in the format the analyzer wants, and results stream back while the audio is still being fed:

let transcriber = SpeechTranscriber(
    locale: locale,
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: [.audioTimeRange]   // timestamps on every run of text
)
// Downloads the on-device model for this language the first time.
if let request = try await AssetInventory.assetInstallationRequest(supporting: [transcriber]) {
    try await request.downloadAndInstall()
}
let format = await SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber])!
 
let analyzer = SpeechAnalyzer(modules: [transcriber])
let (inputs, continuation) = AsyncStream<AnalyzerInput>.makeStream()
 
let collector = Task {
    for try await result in transcriber.results {
        for run in result.text.runs {
            guard let range = run.audioTimeRange else { continue }
            // Spread the run's words across its time range.
        }
    }
}
 
try await analyzer.start(inputSequence: inputs)
while let sample = output.copyNextSampleBuffer() {      // AVAssetReaderTrackOutput
    continuation.yield(AnalyzerInput(buffer: pcmBuffer(from: sample, format: format)))
}
continuation.finish()
try await analyzer.finalizeAndFinishThroughEndOfInput()

One detail mattered for a six-hour file: decoding is far faster than recognition, so the reader would happily push the entire book into memory. The loop pauses whenever it gets more than three minutes of audio ahead of the latest result.

Aligning the transcript with the EPUB

The alignment is deliberately simple. It hashes every four-word window of the ebook, marks hashes that appear twice as unusable, looks up every four-word window of the transcript, and runs a longest increasing subsequence over the matches:

// 1. Four-word sequences that occur exactly once in the ebook.
var unique: [UInt64: Int32] = [:]            // hash -> word index, or -1 if repeated
for i in 0...(ebookTokens.count - 4) {
    let k = key(ebookTokens[i..<i + 4])
    unique[k] = unique[k] == nil ? Int32(i) : -1
}
 
// 2. Every unique sequence the narrator says is a candidate (transcript word i, ebook word j).
for i in 0...(heard.count - 4) {
    guard let j = unique[key(heard[i..<i + 4])], j >= 0 else { continue }
    candidates.append((i, j))
}
 
// 3. Candidates are ordered by time; keep the longest run that also moves forward in the text.
let chain = longestIncreasingSubsequence(candidates, by: \.j)
 
// 4. Drop matches that jump far from both neighbors.
let anchors = chain.filter { isConsistentWithNeighbors($0) }

Recognition errors and unusual names simply produce fewer matches, and a phrase the narrator says that happens to match another chapter gets thrown out because it breaks the chain. Between anchors the app interpolates, and with about two anchors per second of audio that stays within the same sentence.

How the Kindle handoff works

Since the app cannot see my Kindle, the handoff has to be something I can do by hand in a few seconds. A toolbar button opens a small panel with two halves:

• Continue on Kindle shows the chapter, an estimated Kindle percentage, and a short phrase that appears only once in the entire ebook. I type that phrase into the Kindle's search and land on the exact sentence. That phrase comes from the same uniqueness index the sync uses. The percentage is estimated from where the text falls in the EPUB, and Kindle's own number can differ by about one percent.
• Back from Kindle works the other way. I type four or five words from the page I am on, pick the match, and the audiobook jumps there.

It is not magic, but it is reliable, and it is the piece of the app I rely on most.

How it compares to Whispersync and Storyteller

• Whispersync for Voice and Immersion Reading (Amazon) are the polished version of this idea, but only for books bought as a matching Kindle and Audible pair. My audiobooks are files, so they were never an option.
• Storyteller is an open-source, self-hosted project that aligns an EPUB with its audiobook and produces a read-along ebook. It is a good fit if you want the result on many devices. I wanted a native Mac player with no server, and a way back to an unmodified Kindle book.
• Audiobooks keeps the ebook untouched. The sync is a local map between seconds and words, and the Kindle is reached with a search phrase instead of a file format.

The iteration: read or listen, not both

My first instinct after the sync worked was read-along: show the ebook next to the player and highlight the sentence as it is narrated. It looked impressive, but it is not how I use books. When I am listening I am not looking at the screen; when I am reading I do not want a voice in my ear.

So the second pass split the app into two modes that share one bookmark:

• Listen is the player: cover, chapters, progress.
• Read is a quiet, single-column reader for the moments the Kindle is charging or I am at my desk. The paragraph at the top of the page is my bookmark, so switching back to Listen continues from wherever I stopped reading. If I do press play while reading, the page follows the narrator, but that is a side effect rather than the point.

The library got the same treatment. The sidebar became a narrow rail of covers that slides open on hover, so the page gets the full width. Add a focus mode (just the cover and the transport controls) and keyboard shortcuts for everything, and that is the app. It looks like a native Mac app because it is one, and I kept the design deliberately sober. It is the same lesson I learned when I rebuilt Life OS around how I actually used it: build for the habit you have, not the demo.

How Claude Code made the demo video, too

The part I did not expect was the video. I wanted something short to show friends and to put here, and I assumed that meant an evening in a screen recorder and a video editor. Instead I described what I wanted to Claude Code and it worked out the whole production pipeline:

A scripted demo mode inside the app. Recording by hand meant shaky cursor movements and retakes. Claude added a hidden --demo launch flag that sizes the window, hides the cursor, and plays a timed tour on its own. It shows the intro card, opens the library rail, switches to Read, plays the narration, opens the Kindle panel, enters focus mode, and finishes on an outro card. The captions and title cards are drawn by the app itself in SwiftUI, so they match the UI perfectly.

Recording without sound, then rebuilding it. macOS can record a single window cleanly with screencapture, but not the app's audio. So the demo plays muted and writes a log of exactly when playback started and stopped, and at which position in the book. FFmpeg then cuts those exact segments from the original audio file and places them at the matching moments in the video. Claude measured the recording lag (about 0.2 seconds) from the frame where the intro card fades, and corrected for it so the voice lines up with the highlighted sentence.

# Record one window, no shadow, for 56 seconds
screencapture -v -V 56 -o -l"$WINDOW_ID" raw.mov

A vertical cut for phones. Instead of cropping the wide video, which made the text unreadable on a phone, it recorded a second take in a portrait window. It then composited that onto a blurred version of the book cover, with large captions rendered as images above the window.

Choosing the music. I asked it to find music that fit. It went to the royalty-free Apple Loops that ship with GarageBand and built three short candidates: acoustic, synth and solo piano. Since it cannot listen, it compared them on what it could measure, mainly how much energy each one puts in the frequency range of a voice. Then it gave me all three to choose by ear. Its first attempt at lowering the music under the narrator used FFmpeg's sidechain compressor and barely ducked it. It measured that, threw it away, and used a volume envelope instead, since it knew exactly when the narrator speaks:

# Duck the music 80% around each narration window, easing in and out
[0]volume=eval=frame:volume='1-0.8*min(1, clip((t-11.4)/0.6,0,1)*clip((21.2-t)/0.8,0,1) + ...)'[duck];
[duck][1]amix=inputs=2:normalize=0,alimiter=limit=0.9:level=0[a]

The level=0 matters: by default FFmpeg's limiter normalizes its output, which quietly undid the ducking.

Making it copyright-free. The first cut used my own library, which was Frank Herbert's Dune. Fine to send to a friend, not fine to publish. So I asked for a version I could put on this blog. Claude found a Spanish public-domain recording on LibriVox and the matching EPUB on Project Gutenberg, then turned the chapter files into proper M4B audiobooks with chapters and covers. It also added a --library option so the app could open that demo library without touching mine. Then it synced the book and recorded both versions again.

Along the way it reviewed its own output frame by frame and caught a few things before I did: a window title that reappeared on the outro card, playback that started on its own during one take, and a build script still pointing at the old Dune recording.

My role was the same as in my other recent projects, like Mundial Pool and Life OS: deciding what the video should show, giving feedback ("Spanish captions", "a vertical version works too", "no Dune"), and judging the results. The production work, which I would normally have skipped, cost me almost nothing.

Here is the vertical cut:

What I took from it

Building the app was the expected part: a real personal problem, a native stack I can verify, and AI handling most of the typing while I check the behavior. The surprise was how far the same approach stretched past the code. A product demo used to be the thing I never made for side projects because it was a different skill set and a different set of tools. This time it was just another task in the same session, done properly, with the copyright question handled instead of ignored.

The app does exactly one thing I could not get anywhere else: it lets me put the Kindle down mid-sentence and pick the story up in my headphones from that same sentence. More about the app itself is on the Audiobooks project page.