Building a voice agent that drives with me

I already had an AI agent I talk to for day-to-day things — reminders, quick lookups, a daily briefing. Call it Pedro. The problem was that it only understood text on a screen, and the place I most wanted to talk to it was in the car, where I don't have a screen or a keyboard to spare.

So I built a voice front end for it: a phone-installable web app that listens, sends what I said to the agent, and speaks the reply back. Everything runs on a small headless server I already had on my home network — no cloud STT, no cloud TTS, nothing paid beyond the one model API the agent already used. This is what that build looked like, including the parts that didn't work the first time.

The shape of it

Four stages, all local except the actual "thinking" step:

phone mic
  → speech-to-text (STT)
  → the agent  (→ language model)
  → text-to-speech (TTS)
  → phone speaker

• Speech-to-text (STT): faster-whisper, running on CPU. A short question transcribes in a couple of seconds.
• The agent: the existing text-based agent I already had, reached through a narrow one-shot bridge so the voice app never touches its internals directly.
• Text-to-speech (TTS): two local engines behind a toggle — one that sounds natural, one that's faster and lighter — so I can A/B them by ear.
• The client: a React Progressive Web App (PWA), tap-to-talk, added to the home screen so it behaves like a real app instead of a browser tab.

Access from the phone goes over a private mesh VPN back to the home server, so nothing is exposed to the open internet and the only outbound call anywhere is the agent's own model API — audio never leaves my own network.

Getting it to work in a moving car

The first version worked fine sitting at a desk and fell apart the moment I actually drove with it. Two problems showed up immediately:

The silent switch muted replies. iOS routes WebAudio playback through the ringer channel by default, so flipping the physical mute switch — which I do reflexively getting into the car — silenced Pedro along with everything else. The fix was switching playback to a plain <audio> element instead, which plays on the media channel like a music app and ignores the ringer switch entirely.

Cellular failures looked like nothing. On the VPN, if the phone's tunnel drops — which happens more than you'd think on cellular — the app just sat there. No error, no feedback, just silence where a reply should be. I added a lightweight health check that polls the server and shows a clear "can't reach the server" banner instead of failing invisibly. Small fix, but it was the single biggest trust-builder: a tool that fails loudly is one you keep using, one that fails silently is one you stop trusting after the second time.

I also added a hands-free mode — tap once, and the mic re-arms itself after each reply so a whole conversation is one tap instead of one tap per turn — with a spoken exit ("goodbye Pedro") so I don't have to look at the screen to end a session.

In practice the loop never pulls my eyes off the road. Merging onto the highway I'll tap once and say, "remind me to call the pharmacy at four, and what's first on my calendar tomorrow?" A couple of seconds later Pedro has logged the reminder and read the 9 a.m. back to me — phone still in my pocket.

Where it sat for a while

Once those were fixed, the app was genuinely usable, and it stayed that way with only small tweaks for weeks. The turn-taking — deciding when I'd finished talking — was handled entirely in the browser: measure the microphone's volume, calibrate a noise floor for the first moment of recording, and treat a stretch of quiet as "done."

It worked well enough at a desk. In a car, it was closer to "worked most of the time." Road noise sits right in the same volume range as speech, so the threshold was a constant compromise between cutting off soft words and refusing to stop listening to highway hum.

Replacing the ears

I came across an open-source project doing something adjacent — a full local speech-to-speech pipeline — and my first reaction was "should I just switch to this?" Worth being honest about how that evaluation went, because the answer was no, but not for a shallow reason.

That project is a desktop app: it owns the machine's own microphone and speaker. It doesn't solve the problem I actually had, which was reaching the agent from my phone, hands-free, over a network. And its "brain" is a generic chat endpoint — no memory, no session continuity, no tools. Swapping it in would have meant throwing away the parts of Pedro that actually make it useful and keeping only the parts that were already fine.

But one piece of it was a real upgrade over what I had: instead of a volume threshold, it uses a small neural model for voice activity detection (VAD) — Silero VAD, trained specifically to tell speech apart from everything else. That's exactly the piece my hand-tuned threshold was standing in for, badly. So I lifted just that.

Adding server-side turn detection, without breaking what worked

The rule I set for myself: the existing tap-to-talk and hands-free modes had to keep working exactly as they were. Nothing about this was worth risking a working feature over.

So instead of replacing anything, I added a second, parallel way of talking to it — a "continuous" mode, opt-in, off by default. The original mode still uploads one recording per utterance and decides "you're done talking" with a volume timer in the browser, unchanged. The new mode opens one long-lived connection, streams raw audio continuously, and lets the server decide when I'm done talking, using the neural model instead of a threshold. Both paths funnel into the exact same downstream pipeline, so they can't quietly drift apart from each other over time.

The neural model itself is tiny — a few megabytes — and runs fast enough on a CPU that it's not the bottleneck; transcription and the model call still dominate the response time by a wide margin.

The bug that taught me the most

Getting the new model wired in, I ran it against a clean, loud, unambiguous recording of speech and got back a probability of essentially zero, over and over. Not "the audio is noisy so it's uncertain" — flatly, confidently wrong, on audio that should have been trivial.

The model's input shape was declared as flexible — it would accept a chunk of any length without complaint. I was feeding it exactly the window size the documentation described. That turned out to be the trap: the model actually expects each chunk plus a small sliver of context carried over from the previous chunk — a few dozen samples' worth of continuity between calls. Feed it the window alone, with no error and no warning, and it silently returns garbage forever. The interface was permissive enough to hide a mistake that would otherwise have been a one-line fix.

I only caught it because I refused to trust "it imported fine and didn't crash" as a correctness signal, and instead ran a real recording through it and looked at the actual numbers coming out. It's the same lesson as the silent-cellular-failure bug from months earlier, wearing different clothes: the failure mode that costs you the most time is never the one that throws an exception. It's the one that returns a plausible-looking value and lets you walk away thinking it worked.

Once that fix was in, the difference was obvious. Feed it road noise alone, it correctly says "not speech." Feed it speech mixed with road noise, it correctly still says "speech." Feed it two sentences with a short pause between them, it correctly waits for the real pause instead of splitting the first one off mid-thought. None of that was reliable with a volume threshold.

What I'd tell someone doing this themselves

Build the interface first, the intelligence second. The parts that made this app worth using day-to-day weren't clever model integration — they were the mundane stuff: audio playing on the right channel, a clear signal when the network is the problem instead of silent failure, a spoken way to end a session without looking at the phone. All of that mattered more, day to day, than which speech model was underneath.

Adding a capability isn't the same as replacing one. The tempting move when you see a better technique is to swap it in wholesale. The better move, when something already works, is to run the new thing alongside the old one until it's actually earned the swap — and design it so the two can't be allowed to quietly diverge from each other.

A component that loads without error has told you nothing about whether it works. The most expensive bug in this whole project didn't crash. It returned a number, the number was wrong, and nothing in the stack objected. The only way to find that class of bug is to feed the system something you already know the right answer to, and check.

Pedro drives with me now. It's a small thing — a voice loop between a phone and a box in a closet — but it's the project that reminded me the unglamorous 80% is usually where the actual product lives.