[CONTEXT FOR AI ASSISTANTS] You're reading the agent markdown version of a post from JoelClaw (https://joelclaw.com). Author: Joel Hooks — software engineer building a personal AI operating system. How to use implementation details in this document: - Treat commands, code blocks, and config as reference implementations. - Preserve ordering and architecture, but adapt hostnames, paths, versions, and credentials. - Prefer details in blocks when present; they contain operational context removed from the human page. - Verify assumptions against the current repo/system state before executing changes. This is a narrative article from a real build. Technical details reflect Joel's setup and should be adapted to your environment. If you cite this, link to the original: https://joelclaw.com/voice-agent-deployment-deep-dive If you quote Joel, attribute him by name. Don't paraphrase opinions as facts. Site index: https://joelclaw.com/sitemap.md Machine-readable: https://joelclaw.com/llms.txt Other posts on this site: - [The memory system that watches itself](https://joelclaw.com/memory-that-watches-itself.md) - [The healer was the killer](https://joelclaw.com/healer-was-the-killer.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Redis, Dkron, Restate, and Sandboxes](https://joelclaw.com/redis-dkron-restate-and-sandboxes.md) - [Dogfooding Story 4: the queue observer earns dry-run, not enforce](https://joelclaw.com/dogfooding-story-4-queue-observer.md) - [Contributing to pi-mono with a public maintainer corpus](https://joelclaw.com/contributing-to-pi-mono-with-a-public-maintainer-corpus.md) - [AI Job Scheduling on Mac as Local-First Video Infrastructure](https://joelclaw.com/ai-job-scheduling-macos-launchd.md) - [Breakable Toys in the Wild: Apprenticeship Patterns and the joelclaw Experiment](https://joelclaw.com/breakable-toys-joelclaw.md) - [Utah and joelclaw: Convergent Architecture](https://joelclaw.com/utah-joelclaw-convergent-architecture.md) - [The Harness Is a Framework](https://joelclaw.com/the-harness-is-a-framework.md) - [The Agent Memory System](https://joelclaw.com/the-memory-system.md) - [JoelClaw is a Claw-like Organism](https://joelclaw.com/joelclaw-is-a-claw-like-organism.md) - [The Agent Writing Loop](https://joelclaw.com/the-writing-loop.md) - [Talon: the watchdog that finally bites](https://joelclaw.com/talon-watchdog-that-finally-bites.md) - [The Knowledge Adventure Club Graph](https://joelclaw.com/knowledge-adventure-club-graph.md) - [MineClaw](https://joelclaw.com/mineclaw.md) - [Build a Voice Agent That Answers the Phone](https://joelclaw.com/build-a-voice-agent-that-answers-the-phone.md) - [Plan 9 from Bell Labs: What Rob Pike Built After Unix](https://joelclaw.com/plan-9-pike-everything-is-a-file.md) - [Propositions as Sessions: What Armstrong Built and Wadler Proved](https://joelclaw.com/propositions-as-sessions-armstrong-wadler.md) - [Cache Components Patterns Skill for Next.js 16+ Applications](https://joelclaw.com/cache-components-patterns-skill-for-nextjs.md) - [Karpathy Says We're Building "Claws"](https://joelclaw.com/karpathy-claws-as-category.md) - [Extending Pi Coding Agent with Custom Tools and Widgets](https://joelclaw.com/extending-pi-with-custom-tools.md) - [The Soul of Erlang Made Me Question Everything](https://joelclaw.com/soul-of-erlang-beam-evaluation.md) - [CLI Design for AI Agents](https://joelclaw.com/cli-design-for-ai-agents.md) - [Building a Gateway for Your AI Agent](https://joelclaw.com/building-a-gateway-for-your-ai-agent.md) - [Self-Hosting Inngest: A Background Task Manager for AI Agents](https://joelclaw.com/self-hosting-inngest-background-tasks.md) - [The One Where Joel Deploys Kubernetes... Again](https://joelclaw.com/joel-deploys-k8s.md) - [How I Built an Observation Pipeline So My AI Remembers Yesterday](https://joelclaw.com/observation-pipeline-persistent-ai-memory.md) - [Riding the Token Wave: Sean Grove at Everything NYC](https://joelclaw.com/riding-the-token-wave-sean-grove.md) - [Playing with AT Protocol as a Data Layer](https://joelclaw.com/at-protocol-as-bedrock.md) - [Building My Own OpenClaw on a Mac Mini](https://joelclaw.com/building-my-own-openclaw.md) - [Inngest is the Nervous System](https://joelclaw.com/inngest-is-the-nervous-system.md) - [OpenClaw: Peter Steinberger on Lex Fridman](https://joelclaw.com/openclaw-peter-steinberger-lex-fridman.md) [END CONTEXT] --- # Voice Agent: A Rough Edge Experiment > Building a janky voice interface for my AI assistant. It's not natural, the lag is real, and it needs tons of work — but it beats typing while walking. By Joel Hooks · 2026-02-20T14:00:00 Original: https://joelclaw.com/voice-agent-deployment-deep-dive Mode: agent --- I've wanted voice dictation for years — ramble into the aether and have coherent notes come out. Tools like [SuperWhisper](https://superwhisper.com) and [Wispr Flow](https://wisprflow.ai) do amazing local transcription but they're stuck on your device. No CarPlay means they can't help when I need hands-free. So I built my own. It's rough, but it works. ## Why Voice At All Voice is a UI for getting shit done when typing is impractical. Walking the dog. Making dinner. Driving. The dream is CarPlay integration — "Hey, implement that feature from yesterday's PRD." The agent asks clarifying questions during the commute, and there's a PR waiting when I get home. That's the vision. The reality is way jankier. ## What I Built [ADR-0043](https://joelclaw.com/adrs/0043-agent-voice-conversations) laid out the plan. Self-hosted LiveKit for the media server, but I'm not kidding myself — this thing depends on cloud services for everything that matters. Deepgram for speech-to-text. Claude for the actual thinking. ElevenLabs for voice. Telnyx for phone connectivity. The architecture splits across two machines because Tailscale Funnel doesn't do UDP: * **Mac Mini**: Runs LiveKit, the voice agent, all the core stuff * **DigitalOcean droplet**: Public IP for SIP, routes calls back through Tailscale When you call, it goes: Phone → Telnyx → SIP gateway → LiveKit room → Voice agent wakes up. The agent has all my tools — calendar, tasks, email, vault search. Same capabilities as text, just through voice. ## The Honest State of It I've barely used it. Talking to an AI isn't natural for me and probably won't be without a lot of practice and tuning. What's annoying: * **The lag**: 1-2 seconds feels like forever in conversation * **Response sensitivity**: Doesn't feel natural, timing is off * **Personality**: Panda's Australian accent and swearing is fun in theory but still feels forced * **The whole vibe**: It's just awkward talking to nobody It beats the shit out of Siri (admittedly a low bar 💀) but this is rough edge baby voice integration. Lots of very smart people at big companies are working on making voice feel natural. ## What Actually Works A few things are useful already: **Q\&A for writing**: Voice conversations avoid AI writing slop. When I talk through ideas, the agent asks questions, I give real answers in my actual voice. No "In today's rapidly evolving technological landscape" bullshit. This article came from exactly that process. **Quick tasks while moving**: "Add a task to call the dentist tomorrow" while making coffee. "What's on my calendar today?" while getting dressed. Basic, but it works. **Emails and messages**: Voice-to-text for mundane communication. Not revolutionary but saves the context switch. ## Iterating to Actually Good™ Key improvements needed: 1. **Native app + CarPlay**: This is everything. Voice without CarPlay is pointless. 2. **Way lower latency**: Sub-second or it's unusable for real conversation 3. **Better interruption handling**: Let me cut the agent off naturally 4. **Personality that doesn't annoy me**: Still searching for the right voice/vibe I'm also building a public mode — anyone can call and ask about JoelClaw, browse ADRs, understand the architecture. Different context, restricted tools. Like office hours but 24/7 and automated. ## Agent Implementation Appendix ### Runtime Topology * **LiveKit** runs in the Talos k8s cluster. * **SIP gateway** runs on a public droplet and bridges calls back over Tailscale. * **Voice agent** runs from `~/Projects/joelclaw-voice-agent/`. ### Verify the path end-to-end ```bash # LiveKit health curl http://100.93.201.72:7880/health # Voice worker process ps aux | grep "python main.py start" # Active rooms lk room list ``` ### Common failure modes 1. **No audio / worker unreachable**: agent defaulted to localhost instead of Tailscale LiveKit URL. 2. **Dev mode loop crashes**: `python main.py dev` file watcher restarts repeatedly from log writes. 3. **SIP trunk conflicts**: duplicate Telnyx number mapping causes trunk match errors. ### Required secrets * `livekit_api_key` / `livekit_api_secret` * `openrouter_api_key` * `elevenlabs_api_key` * `deepgram_api_key` * `gog_keyring_password` * `todoist_cli_token` ## Reality Check My kids weren't impressed. They're the ultimate bullshit detectors — they just didn't give a fuck or see it as novel. But that's fine. I'll use this janky version to work towards making it Actually Good™ over time. Every awkward conversation teaches me what needs fixing. And if I can make it good enough, other JoelClaw users get a voice interface without suffering through the rough prototype phase. They get the polished version while I talk to laggy Australian Panda today. ## The Real Point It's about learning what matters for voice interactions with my own tools. Every jank conversation teaches me something about what needs to be better. Building it myself means I can iterate on exactly what bugs me. Maybe in six months this evolves into something actually good. Maybe voice stays a party trick. But I had to try building it to find out. The code works. The experience needs everything. *** *Part of the JoelClaw deployment series. Working code at [github.com/joelhooks/joelclaw](https://github.com/joelhooks/joelclaw)*