A transcript without speaker names is hard to use. If an audio file has two people talking, I need to know which words came from which person. My pipeline can label speakers automatically. The question was whether I could trust the labels.
I did not want to trust them just because the output looked clean. So I made an answer key by hand, and I let the computer grade itself against it.
The Need: “Is the Speaker Label Right?” Has No Obvious Answer
The pipeline runs on a GPU server that someone lent me access to (an NVIDIA GB10 machine). It does three things:
- faster-whisper turns the audio into text with time stamps.
- pyannote cuts the audio into “who spoke when” pieces.
- A small script puts the two together, so each sentence gets a speaker label.
The labels looked reasonable when I skimmed them. But “looks reasonable” is not a test. If I later build anything on top of these labels, such as a summary of what one person said, a wrong label quietly makes the result wrong, and nobody can see why.
The file I used is a 39-minute audio recording with two speakers. It is test data.
The Proof: 91.5% by Segment, 93.8% by Word
I listened to the audio and marked the stretches where Speaker 1 was talking. A small tool I already had saved those marks to a JSON file. Then a script compared my marks with the AI’s labels.
| Measure | Result |
|---|---|
| Match, sentence by sentence (what the pipeline does now) | 91.5% (88.0% before the time-offset fix below) |
| Match, word by word | 93.8% (91.1% before the fix) |
| Speaker 1 detection, precision / recall (word level) | 94.7% / 93.4% |
| Whisper segments that contain two speakers | 50 of 451 (11%) |
| Speaker pieces pyannote produced | 1,066, for a 39-minute file |
| Stretches where my marks and the AI disagreed | 79 |
My honest reading: it was better than I feared and worse than I would like. About one sentence in twelve gets the wrong label. Most of the loss comes from sentences that contain a change of speaker. Whisper cuts text by pauses, pyannote cuts audio by voice, and the two don’t always cut in the same place.
I have not changed the production code yet. Word-by-word labeling is the better method, and I checked that it works. But I only checked it on this one file.
The Story: I Was Short on Tokens, So I Changed Who Does the Work
I use a Claude subscription, and that week my token allowance was nearly used up. That changed how I did this job.
Two decisions came from that:
- Run only one file. Re-transcribing with time stamps takes a long GPU run, and the old results had no time information anyway. One file was enough to learn whether the idea works.
- Don’t let Claude read the comparison. I asked for the comparison to be done by the local Qwen model on the GPU box. Claude would write the script once, run it, and save the results to a file. It would not read the output.
There is one more detail I liked. After the comparison, I asked for a version of the transcript with “Speaker 1” and “Speaker 2” labels applied from my marks, and I said to use Qwen for it. Claude said no to that part, politely, and used a plain script. Its reason was simple. Applying a label means checking whether each word’s time falls inside a marked stretch. That is arithmetic. A language model can get the times wrong or quietly change a word. A script can’t.
I think that is a good rule for anyone doing this: use the language model for opinions, and use a script for arithmetic.
One more thing about my marks. I pressed a key while listening, so my marks arrive a little after the real speech starts. The comparison script estimates that delay and shifts my marks back. That one correction moved the sentence-level match from 88.0% to 91.5%. It also reminds me that my “answer key” is not perfect. It is only better than the thing it is grading.
The How: A Hand-Made Answer Key in Four Steps
Tools: faster-whisper (word time stamps on), pyannote for speaker pieces, a local Qwen model through Ollama for the written commentary, plain Python for the numbers. Cost: $0 in API fees. The only cost was my token allowance.
- Mark the truth by ear. Play the audio and mark the spans where Speaker 1 talks. Save them as
marks.json: a list of start and end times in seconds. - Re-transcribe with word times. Run the pipeline so it keeps the time of every word, not only each sentence. This is needed for the word-level score. Old results without times can’t be compared.
- Compare with a script. For each word, ask two questions: does its time fall inside a marked span, and what did the AI label it? Count the matches. Try small time shifts for my key-press delay and keep the shift that gives the best match.
- Ask the local model for commentary, not numbers. Send it only the list of the worst mismatches and ask for the type of error, a guess at the cause, and a possible fix. The script also saves the 40 worst stretches with their times, so I can go listen to them myself.
💬 Prompt that worked “The marks.json file in my Downloads folder marks where Speaker 1 talks in that audio file. The automatic speaker split is not accurate, so I listened and marked the positions myself. Can you check whether linking this file to the existing transcript can improve the speaker separation?”
💬 Prompt that worked “Compare the marks file with the transcript, to compare the speaker split I checked myself with the one from the GPU box. Don’t do it yourself. Run this comparison with the Qwen model on the GPU box. I need to save tokens.”
🗂 Claude.md Rule Arithmetic on time stamps goes to a script, not to a language model. Use the model only for commentary on the results.
What I Still Don’t Know
I haven’t judged how good Qwen’s written commentary is. It ran without errors, and the file exists, but I haven’t checked its advice against the 40 stretches yet. One file also proves little. A recording with more people talking over each other could score much lower.
Next, I want to apply the word-level method to the real pipeline and re-run this same file, so I can see whether the labels improve in practice.