Real-Time Translation

I built TransLee for my own family, using the same Gemini Live API that Google publishes starter code for. That raises an obvious question I hadn’t actually answered yet: does something like this already exist, built by someone else, that I could have just used? So I searched GitHub for it. What’s Already Out There A few projects showed up using the same Gemini Live API for translation: kazunori279/live-translator — three modes: an agent mode, a simultaneous-translation mode, and a conversation mode for two people talking to each other. Built by a Google Cloud developer relations engineer. google-gemini/gemini-live-translate-livekit and livekit-examples/gemini-live-translate — official Google examples for broadcasting one speaker’s voice to many listeners, each hearing their own language, built on LiveKit. kazunori279/adk-live-translator, noodleA1/live_translation, folubebe/gemini_realtime_speech_to_text — smaller projects, closer to learning exercises than finished apps. The Feature I Thought Was Unique, Wasn’t TransLee has a “translated voice” checkbox for its conversation mode. Check it before you speak, and your Korean gets translated and spoken out loud in whatever language you picked, so you can take part in a meeting in a language you don’t speak. I wrote about how that checkbox came to be in an earlier post in this series — it took fourteen discarded attempts at automatic detection before I realized the checkbox already did the job. ...

TransLee could already do two separate things well: catch sound from a phone’s own speaker and translate it, or listen to a person talking and speak a translation back. What it couldn’t do was the thing I actually needed for a real meeting: sit in an English-language call as the only Korean speaker, show me Korean for everything everyone else says, and speak English out loud when it’s my turn, without me touching a language dropdown mid-conversation. ...

Late on July 13, TransLee could hear a quiet news broadcast through my phone’s own speaker and translate it live. By the next morning, that exact feature was silent. Nothing had touched the speaker-recognition code overnight. What had changed was a different feature entirely: translated voice playback had just started working for the first time. The Symptoms, in Order 03:40 — Speaker-sound recognition, which worked the night before, stopped picking up anything. 04:36 — Still broken. YouTube audio wasn’t recognized at all, and even the transcript-only fallback wasn’t translating, just showing raw text. 07:41 — Fixed the transcript-and-translation display. Recognition worked again. But the voice-playback feature, the one that had worked the previous evening, was now silent. 07:46 to 08:13 — Every attempt to get both working at once produced the same shape of failure: one sentence would translate and play correctly, then everything stopped, no more translation, no more transcript, sometimes a garbled half-second of audio right before it died. Five hours of this, with the failure mode staying identical no matter which side I patched. That repetition was the actual clue. This wasn’t a bug hiding somewhere in the code. It was two features fighting over the same microphone. ...

I want to say this plainly before I say anything else: Google’s engineers built Gemini Live Translate. Google published the starter code for it. I did not invent real-time speech translation. What I actually did is narrower, and honestly more interesting to write about: I took a script Google gives away for free and turned it into a service I could open from my own phone, on my own domain, before I had confirmed Google’s own consumer app was reliably doing the same thing for me here in Korea. ...

The Korean tagline I wrote for the original Play Store listing translates roughly to: “Even a whisper gets translated, reading YouTube by ear.” It was a promise about one specific feature, not a slogan I picked for how it sounded. What “Listening to Its Own Speaker” Actually Means Most translation apps expect you to talk into them, or hold them up to someone else talking. TransLee has always done something slightly different: it can also just sit next to whatever sound is already in the room, a phone playing a YouTube video, a TV, a radio, and translate that sound live, without any screen-sharing or app integration on the source side. ...

The old version of TransLee worked. People used it. It just made you wait. The Old Pipeline: Record, Wait, Stitch The original app was built in Flutter, with a Node.js server in the middle talking to OpenAI’s Whisper for speech-to-text and GPT for translation. The flow looked like this: [Record 5-15 seconds] -> [Whisper: speech to text] -> [GPT: translate text, using the tail of the previous chunk for context] -> [Text-to-speech] -> [Play the translated audio] ⬑ repeat, waiting the full cycle each time ⬏ Five to fifteen seconds isn’t a random number. It was a setting I chose on purpose, because shorter chunks meant the translation kept losing the thread mid-sentence, and longer chunks meant a longer wait before you heard anything at all. The fix for the “losing the thread” problem was to keep the last bit of the previous chunk’s text and glue it onto the front of the next one before sending it off to be translated, so GPT had enough context to know what was going on. It worked. It also meant every single sentence arrived several seconds after it was actually spoken. ...

I put my phone next to a TV playing a Kenyan news broadcast in Swahili, low volume. My translator picked up the sound and started speaking the English translation back to me as the newscaster kept talking. No 5-second wait. No 15-second wait. The words just kept coming. It’s a Progressive Web App: open the link on a phone, add it to the home screen, and it behaves like a native app, no App Store or Play Store involved. ...