TransLee could already do two separate things well: catch sound from a phone’s own speaker and translate it, or listen to a person talking and speak a translation back. What it couldn’t do was the thing I actually needed for a real meeting: sit in an English-language call as the only Korean speaker, show me Korean for everything everyone else says, and speak English out loud when it’s my turn, without me touching a language dropdown mid-conversation.

The Need

Picture an actual meeting. It’s in English. I’m Korean. I want two things running at once, automatically:

  • Whatever anyone else says, I want to read in Korean on my screen.
  • Whatever I say in Korean, I want the app to speak out loud in English, so the other people in the call can understand me without looking at my phone.

Those are two different translation directions, running in the same conversation, and the app has no way to know which direction applies to any given moment except by knowing who’s talking. That’s the whole problem this post is about.

The Proof

As of version 1.5, it works the way I described above. In “conversation interpretation” mode, listening is always translated to Korean text. When I check the “translated voice” box before I talk, my Korean speech gets translated and spoken out loud in whatever language I’ve picked, English by default, out of about 70 languages. Uncheck it, and it goes back to just listening and showing Korean text for whoever’s talking. That’s the whole interaction: check it when it’s my turn, uncheck it when it isn’t.

It sounds almost too plain to be the outcome of a full day of work. It is. That’s most of what this post is actually about.

The Story: Fourteen Versions That Never Shipped

On July 15, I spent most of a day trying to make this automatic: detect, from the incoming audio and text alone, whether the person currently talking was me or the other side, and switch translation direction without anyone touching a checkbox.

The git history for TransLee’s web app has exactly two commits from that stretch, an old v1.2 and, at the end of the day, v1.5. In between, according to my own project notes, there were fourteen numbered attempts, an old v1.3 through v1.16, that never made it into a commit. Every one of them got tried, failed in actual use, and got thrown away before the day was over.

A rough shape of what got tried and why each one broke:

  • Reconnect mid-session to a guessed third language. The Live API’s translation target is set once, when the connection opens. To auto-switch direction, I had the app tear down and reopen the connection with a new guess every time it thought the speaker had changed. Transcripts started coming back mangled, an English sentence like “my name is” would come back transcribed as if it were Korean, phonetically, “마이 네임 이즈.” Reconnecting a live translation session that often, based on a guess, made the model unstable.
  • A documented “don’t echo the target language back” flag. The API has an echoTargetLanguage option, meant to stop the model from just reading back audio in a language it was already given. I turned it on. It made no measurable difference on its own.
  • Comparing the original and translated text for an exact match. If the source text and the translated text came back identical, that was treated as a sign the model was hearing its own target language and had nothing to translate, so mute the voice. This missed real cases, “Nice to meet you” transcribed with just the word “Nice” mangled broke an exact-match check, and it let audio leak through it shouldn’t have.
  • Word-overlap detection instead of exact match. Loosen the check to “60% of the words match” instead of “identical strings.” This caught more of the leaks, and for a while it looked promising, one test run correctly muted “Nice to meet you” a few seconds in. But the same run also let two later, nearly identical sentences through unmuted, and the timing kept slipping depending on how fast the reply arrived relative to when the check ran.

None of these were bad ideas on paper. Each one also introduced a new, different kind of glitch, and fixing that glitch usually broke whatever the previous version had gotten right. By the afternoon I’d cycled through session-reconnect attempts, an API flag, exact-text matching, word-overlap matching, and a delayed-playback queue to buy the detection more time, and every version had its own way of failing in a live conversation.

The Insight

The turn came less from a code change than from stepping back. TransLee already had a “translated voice” checkbox. Checking it on meant “I’m speaking now, translate and speak it.” All the detection work I’d been doing for a day was trying to teach the app to guess the exact same fact automatically, from unreliable signals, when the actual fact, who’s talking right now, was something I already knew and could just tell it.

💬 Prompt that worked “I just realized something. Muting when the recognized language equals the target language was pointless all along. Unchecking the ’translated voice’ box already does that. If I want sound when I’m the one talking in a meeting, I check the box, and uncheck it when I’m done. I basically just spent a day automating something a checkbox already did perfectly, and given how unstable that automation turned out to be, it wasn’t worth it.”

That single realization ended the whole line of automatic detection. But it wasn’t quite the full answer yet. A checkbox that means “my turn to talk” still needs the app to know two things it hadn’t cleanly separated: what to show when I’m listening, and what to speak when I’m talking. Those turned out to be two different settings wearing one label.

The Reframe

The version right before the checkbox insight used a single “target language” setting for both directions, and that’s what caused the last real bug: if I left that setting on its default of Korean and then checked the box to speak, the app tried to translate Korean into Korean. Nothing came out, no text, no voice, because there was nothing to translate.

The fix was to stop thinking of it as one target language and split it into two:

💬 Prompt that worked “What about this: by default, translate whatever foreign language I hear into Korean. When I check the translated-voice box, translate my Korean into English instead. Apply this only in conversation interpretation mode; speaker mode stays exactly as it is now.”

That’s the shape that shipped as v1.5. Listening always targets Korean, fixed, one option. Speaking targets a separate language, defaulting to English, chosen from about 70 options. Since the two settings can never hold the same value by construction, the “translating Korean into Korean” bug can’t recur. The recognized-language dropdown got simpler too, down to auto-detect or English, since it no longer had to carry any of the direction logic.

The How

The mechanism, once it settled, is small. effectiveTargetLang() returns Korean unless conversation mode is active and the voice checkbox is checked, in which case it returns whatever language is picked for speaking. Checking the box, unchecking it, changing the mode, or changing the speak-language while mid-turn each call reconnectForNewTarget(), which finalizes whatever’s currently on screen and reopens the Live API connection with the new target.

🗂 Claude.md Rule Reconnecting mid-session to change the Live API’s translation target isn’t the problem — guessing when to do it from real-time text is. A reconnect triggered by an explicit user action, switching only between the two languages actually in use, is stable. The same reconnect triggered automatically, dozens of times a minute, from an unstable guess about who’s talking, breaks the model: transcripts turn into phonetic mush and hallucinations increase.

Why this reconnect is safe when all fourteen earlier ones weren’t: it fires rarely, roughly once per conversational turn instead of on every scrap of incoming text, it’s triggered by something 100% accurate (I clicked it) instead of a guess, and it only ever switches between the two languages actually in play, Korean and whatever I picked for speaking, never to some third language a detector invented. If I try to set the speaking language to Korean too, matching the listening language, the app shows a warning instead of quietly breaking.

I also tried, and abandoned the same day, a separate idea: let me tap a transcript line and get an on-demand translation for just that sentence, instead of trying to mute or unmute automatically. It sounded reasonable until I actually tried using it in a live conversation. By the time I’d tapped a line and read the translation, the meeting had moved three sentences further along. A real-time conversation doesn’t wait for you to go check something. That one didn’t survive contact with an actual use case either, and it’s not in the app today.

No extra API cost came out of any of this. The reconnects use the same Live API connection the app already needed; nothing here calls a separate translation endpoint. The cost of the day was entirely my own time spent trying, and discarding, roughly a dozen ways to automate a decision that a checkbox was already answering.