WhisperX: Free, Local Transcription for Audio That Can't Leave the House

Not every recording I transcribe is a Bible teaching or a home video. Some of it is confidential — calls from a counseling hotline, where people are talking about the hardest thing in their life. That kind of audio can’t go anywhere near a paid cloud API. Not because of cost. Because it isn’t mine to send anywhere.

That ruled out Whisper’s API and ElevenLabs — the two services I’d already used and paid for — for this particular kind of file. I needed something that would run entirely on my own PC, touch nothing outside it, and cost nothing per file. My PC has no GPU. Whatever I built had to live with that too.

Two Kinds of Audio, Two Sets of Rules

I split what I transcribe into two buckets:

• General topics — sermons, talks, home recordings. No privacy issue. These can go through any cloud model, local or not.
• Counseling calls — confidential by nature. These never leave my machine, full stop.

💬 Prompt that worked “My audio falls into two groups: confidential counseling calls, and general topics with no privacy concern. The general ones can use big cloud models. The counseling ones have to run locally, and I don’t have a GPU. Can I make up for that gap somehow?”

That question is what shaped everything that followed — one tool, two paths through it, depending on which bucket a file falls into.

Why WhisperX

WhisperX is an open-source wrapper around OpenAI’s Whisper model that adds word-level alignment and speaker diarization on top of it. The part that mattered to me: it runs entirely on your own machine. No API key for the transcription itself, no upload, no per-minute charge.

The catch is that Whisper models — even the smaller ones — are built with a GPU in mind. My PC (a Ryzen 5 laptop, no dedicated GPU) has to fall back to plain CPU:

DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
COMPUTE_TYPE = 'float16' if DEVICE == 'cuda' else 'int8'

On CPU it uses int8 quantization instead of float16 — a little less precision, in exchange for something that actually finishes in a reasonable time. On a 5.75-minute test recording, that came out to about 7.2 minutes without speaker separation, and 8.3 minutes with it. Slower than real time, but free, private, and running on hardware I already owned.

The Low-Spec Mode

That CPU number is the good case. The machine I actually worry about is a Lenovo IdeaPad I bought at Costco earlier this year for about ₩430,000 (still on the shelf there as of this writing) — an Intel N150, 8GB of RAM (7.68GB usable), integrated graphics only. On a machine that small, the default settings crash outright. WhisperX’s default batch size (8) tries to encode several audio segments at once, and if there isn’t enough memory for that, the process just dies mid-run.

The fix was a second mode: skip word-level alignment and drop the batch size to 2.

if no_align:
    cmd.append('--no_align')
    cmd += ['--batch_size', '2']

You lose word-level timestamps and speaker diarization in this mode — you get sentence-level chunks instead — but the job finishes instead of crashing. On a low-RAM machine, a slower finished transcript beats a fast crash, every time.

Two Gotchas That Each Cost an Evening

Korean text killed the subprocess. WhisperX runs as a separate process, and Windows terminals default to the cp949 codepage. The moment WhisperX tried to print Korean text through that pipe, the whole thing died with a UnicodeEncodeError — not a transcription bug, just an encoding mismatch between two processes talking to each other.

🗂 Claude.md Rule Subprocesses that print Korean (or any non-ASCII) text on Windows need PYTHONUTF8=1 and PYTHONIOENCODING=utf-8 set explicitly in the child process’s environment. The console’s default cp949 codepage will kill the process otherwise — this has bitten this project more than once.

A token was required for a feature I’d turned off. Speaker diarization needs a Hugging Face token. Fair enough — except the token turned out to be required even with diarization unchecked, because the voice-activity-detection step underneath uses the same pyannote pipeline either way. Turning off diarization doesn’t remove the dependency; it’s baked into an earlier stage I hadn’t noticed.

Cleaning Up What Whisper Gets Wrong

Two more problems showed up once transcripts started coming back.

Hallucination. In silent or noisy stretches, Whisper sometimes locks onto a phrase and repeats it dozens of times. The fix isn’t preventing it — I haven’t found a way to — it’s catching the pattern afterward and collapsing it into one line with a note: (same phrase repeated 12 times, collapsed). Ordinary repetition (“yes, yes, yes”) stays untouched; only runs of five or more get flagged.

Wrong names. Person names, place names, organization names — anything uncommon gets mangled fairly often. I looked for a transcription model already trained for counseling-call audio specifically. There isn’t one that I could find. Instead, I built a glossary: a plain text file of correct terms, fed to WhisperX as --hotwords during transcription, and fed again to the correction step afterward as a hint. It isn’t model training. It’s just telling the same tool, twice, what it tends to get wrong.

Two Correction Paths, Matching the Two Buckets

Transcription is only half the job — raw output still needs cleanup. I ended up with four correction engines, split along the same privacy line as the audio itself:

The rule is simple: confidential audio never touches the two cloud rows. Everything else can, because there’s nothing at risk in sending it.

One dead end along the way: I tried Qwen3’s smaller “thinking” models for correction first. There’s no way to turn thinking mode off — not through the API parameter, not through prompt tricks like /no_think. The model would chew on a single paragraph for minutes and sometimes wander off-topic entirely.

🗂 Claude.md Rule Qwen3 “thinking” models can’t be forced into non-thinking mode via API parameter or prompt trick. If a task needs fast, deterministic output, switch to a genuinely non-thinking model (qwen2.5:3b-instruct worked here) rather than fighting Qwen3’s reasoning mode.

What It Costs

For the counseling calls, the whole pipeline runs for $0. WhisperX, ET5, and Qwen2.5 are all free, local, and running on hardware I already owned. The two paid engines — DeepSeek and Gemini — never touch that audio at all.

For general audio, the cloud engines are pay-per-use, same as the API costs I’ve written about before. But that’s a choice, not a requirement. The free local path works on any audio — it’s just slower, and rougher on names it doesn’t recognize.

What’s Still Rough

The ET5 engine occasionally doubles up punctuation (?.) — unfixed for now. On badly garbled stretches, it sometimes “corrects” things into stranger phrasing than it started with. And the honest answer to “shouldn’t there be a model trained specifically for this kind of transcription” is: probably, but I couldn’t find one, and training one myself would take a GPU and labeled data I don’t have. The glossary is a workaround, not a solution.

None of that changes the one thing that had to be true: nothing confidential leaves this machine. Everything else is a speed-and-polish problem I can keep chipping away at.