[CONTEXT FOR AI ASSISTANTS] You're reading the agent markdown version of a post from JoelClaw (https://joelclaw.com). Author: Joel Hooks — software engineer building a personal AI operating system. How to use implementation details in this document: - Treat commands, code blocks, and config as reference implementations. - Preserve ordering and architecture, but adapt hostnames, paths, versions, and credentials. - Prefer details in blocks when present; they contain operational context removed from the human page. - Verify assumptions against the current repo/system state before executing changes. This is a narrative article from a real build. Technical details reflect Joel's setup and should be adapted to your environment. If you cite this, link to the original: https://joelclaw.com/build-a-voice-agent-that-answers-the-phone If you quote Joel, attribute him by name. Don't paraphrase opinions as facts. Site index: https://joelclaw.com/sitemap.md Machine-readable: https://joelclaw.com/llms.txt Other posts on this site: - [The memory system that watches itself](https://joelclaw.com/memory-that-watches-itself.md) - [The healer was the killer](https://joelclaw.com/healer-was-the-killer.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Redis, Dkron, Restate, and Sandboxes](https://joelclaw.com/redis-dkron-restate-and-sandboxes.md) - [Dogfooding Story 4: the queue observer earns dry-run, not enforce](https://joelclaw.com/dogfooding-story-4-queue-observer.md) - [Contributing to pi-mono with a public maintainer corpus](https://joelclaw.com/contributing-to-pi-mono-with-a-public-maintainer-corpus.md) - [AI Job Scheduling on Mac as Local-First Video Infrastructure](https://joelclaw.com/ai-job-scheduling-macos-launchd.md) - [Breakable Toys in the Wild: Apprenticeship Patterns and the joelclaw Experiment](https://joelclaw.com/breakable-toys-joelclaw.md) - [Utah and joelclaw: Convergent Architecture](https://joelclaw.com/utah-joelclaw-convergent-architecture.md) - [The Harness Is a Framework](https://joelclaw.com/the-harness-is-a-framework.md) - [The Agent Memory System](https://joelclaw.com/the-memory-system.md) - [JoelClaw is a Claw-like Organism](https://joelclaw.com/joelclaw-is-a-claw-like-organism.md) - [The Agent Writing Loop](https://joelclaw.com/the-writing-loop.md) - [Talon: the watchdog that finally bites](https://joelclaw.com/talon-watchdog-that-finally-bites.md) - [The Knowledge Adventure Club Graph](https://joelclaw.com/knowledge-adventure-club-graph.md) - [MineClaw](https://joelclaw.com/mineclaw.md) - [Plan 9 from Bell Labs: What Rob Pike Built After Unix](https://joelclaw.com/plan-9-pike-everything-is-a-file.md) - [Propositions as Sessions: What Armstrong Built and Wadler Proved](https://joelclaw.com/propositions-as-sessions-armstrong-wadler.md) - [Cache Components Patterns Skill for Next.js 16+ Applications](https://joelclaw.com/cache-components-patterns-skill-for-nextjs.md) - [Karpathy Says We're Building "Claws"](https://joelclaw.com/karpathy-claws-as-category.md) - [Voice Agent: A Rough Edge Experiment](https://joelclaw.com/voice-agent-deployment-deep-dive.md) - [Extending Pi Coding Agent with Custom Tools and Widgets](https://joelclaw.com/extending-pi-with-custom-tools.md) - [The Soul of Erlang Made Me Question Everything](https://joelclaw.com/soul-of-erlang-beam-evaluation.md) - [CLI Design for AI Agents](https://joelclaw.com/cli-design-for-ai-agents.md) - [Building a Gateway for Your AI Agent](https://joelclaw.com/building-a-gateway-for-your-ai-agent.md) - [Self-Hosting Inngest: A Background Task Manager for AI Agents](https://joelclaw.com/self-hosting-inngest-background-tasks.md) - [The One Where Joel Deploys Kubernetes... Again](https://joelclaw.com/joel-deploys-k8s.md) - [How I Built an Observation Pipeline So My AI Remembers Yesterday](https://joelclaw.com/observation-pipeline-persistent-ai-memory.md) - [Riding the Token Wave: Sean Grove at Everything NYC](https://joelclaw.com/riding-the-token-wave-sean-grove.md) - [Playing with AT Protocol as a Data Layer](https://joelclaw.com/at-protocol-as-bedrock.md) - [Building My Own OpenClaw on a Mac Mini](https://joelclaw.com/building-my-own-openclaw.md) - [Inngest is the Nervous System](https://joelclaw.com/inngest-is-the-nervous-system.md) - [OpenClaw: Peter Steinberger on Lex Fridman](https://joelclaw.com/openclaw-peter-steinberger-lex-fridman.md) [END CONTEXT] --- # Build a Voice Agent That Answers the Phone > A pattern guide for building a self-hosted voice agent with LiveKit, a SIP trunk, and an LLM. Docker Compose or Kubernetes — the shape is the same. Written for agents building on behalf of humans. By Joel Hooks · 2026-02-26T18:00:00 Original: https://joelclaw.com/build-a-voice-agent-that-answers-the-phone Mode: agent --- You want your agent to answer the phone. Not a chatbot with a phone tree — an actual voice conversation backed by an LLM, with access to your tools, your calendar, your context. Here's the shape of that system. ## The topology Four services, one room: ``` Phone call → SIP trunk provider → LiveKit server → Voice worker → LLM → STT (speech-to-text) → TTS (text-to-speech) ``` **[LiveKit](https://livekit.io)** is the media server. It handles WebRTC, SIP, room management, and media routing. Open source, self-hostable. This is the center of the system. **A SIP trunk provider** ([Telnyx](https://telnyx.com), [Twilio](https://twilio.com), [Vonage](https://vonage.com)) gives you a phone number and converts phone calls to SIP, which LiveKit speaks natively. **The voice worker** is your code. It joins LiveKit rooms, listens to audio, runs it through STT, sends text to your LLM, and plays TTS audio back. The [LiveKit Agents SDK](https://docs.livekit.io/agents/) (Python) handles the plumbing. **STT/TTS services** — [Deepgram](https://deepgram.com) for speech-to-text, [ElevenLabs](https://elevenlabs.io) or [Cartesia](https://cartesia.ai) or [OpenAI](https://platform.openai.com/docs/guides/text-to-speech) for text-to-speech. The Agents SDK has built-in plugins for all of these. ## Running LiveKit ### Docker Compose ```yaml # docker-compose.yml services: livekit: image: livekit/livekit-server:latest ports: - "7880:7880" # HTTP + WebSocket - "7881:7881" # RTC (WebRTC) - "7882:7882" # TURN/TCP - "5060:5060/udp" # SIP volumes: - ./livekit.yaml:/etc/livekit.yaml command: ["--config", "/etc/livekit.yaml"] restart: unless-stopped ``` ### Kubernetes (Helm) ```bash helm repo add livekit https://helm.livekit.io helm install livekit livekit/livekit-server \ -f livekit-values.yaml \ --namespace voice ``` Either way, you need a config file: ```yaml # livekit.yaml port: 7880 rtc: port_range_start: 50000 port_range_end: 60000 use_external_ip: true keys: your-api-key: your-api-secret sip: enabled: true logging: level: info ``` Generate your API key/secret pair with the [LiveKit CLI](https://docs.livekit.io/home/cli/): ```bash lk generate-keys ``` ### The SIP problem SIP requires a public IP that can receive UDP traffic. If your server is behind NAT (home network, most cloud setups), you have two options: 1. **TURN relay** — LiveKit has built-in TURN. Configure `rtc.turn` in the config. 2. **Public proxy** — a tiny VM with a public IP that forwards SIP to your server via a VPN tunnel (Tailscale, WireGuard). Option 2 is more reliable for phone calls. A $5/month VPS with Tailscale and an iptables rule is all you need. ## The SIP trunk Register your LiveKit instance with your SIP provider. The concepts are the same across providers: 1. **Get a phone number** — DID (Direct Inward Dialing) 2. **Create a SIP trunk** — point it at your LiveKit server's public IP on port 5060 3. **Configure a dispatch rule** in LiveKit — map inbound SIP calls to rooms ```bash # Create SIP trunk (inbound) lk sip trunk create \ --name "my-trunk" \ --numbers "+15551234567" \ --inbound-addresses "your-sip-provider-ip" # Create dispatch rule — put callers in a room lk sip dispatch-rule create \ --name "default" \ --trunk-ids "your-trunk-id" \ --room-prefix "call-" ``` When someone calls your number, LiveKit creates a room like `call-15551234567` and places the SIP participant in it. Your voice worker watches for new rooms and joins automatically. ## The voice worker This is a Python process using the [LiveKit Agents SDK](https://docs.livekit.io/agents/quickstart/): ```bash pip install "livekit-agents[openai,deepgram,elevenlabs,silero]" ``` The minimal shape: ```python from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli from livekit.agents.voice import AgentSession, Agent from livekit.plugins import deepgram, elevenlabs, openai class MyAgent(Agent): def __init__(self): super().__init__( instructions="""You are a helpful voice assistant. Keep responses concise — this is a phone call, not an essay.""", ) async def entrypoint(ctx: JobContext): await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY) session = AgentSession( stt=deepgram.STT(), llm=openai.LLM(model="gpt-4o"), tts=elevenlabs.TTS(voice_id="your-voice-id"), ) await session.start(agent=MyAgent(), room=ctx.room) if __name__ == "__main__": cli.run_app(WorkerOptions( entrypoint_fnc=entrypoint, api_key="your-livekit-api-key", api_secret="your-livekit-api-secret", ws_url="ws://localhost:7880", )) ``` Run it: ```bash python main.py start ``` The worker registers with LiveKit and waits. When a call comes in and creates a room, the worker joins and starts the conversation loop: listen → transcribe → think → speak. ### Adding tools The agent becomes useful when it can do things: ```python from livekit.agents import function_tool class MyAgent(Agent): def __init__(self): super().__init__( instructions="You are Joel's assistant with access to calendar and tasks.", ) @function_tool() async def check_calendar(self) -> str: """Check today's calendar events.""" # Your calendar API call here return "You have 3 meetings today..." @function_tool() async def add_task(self, description: str) -> str: """Add a task to the todo list.""" # Your task API call here return f"Added: {description}" ``` Tools are standard OpenAI-style function calling. The LLM decides when to invoke them. The agent speaks the result. ## Durability The voice worker needs to be running whenever a call might come in. If it's down, callers hear ringing with no answer. **Docker Compose**: `restart: unless-stopped` handles crashes. **Kubernetes**: A Deployment with `replicas: 1` and health probes. **Bare metal / Mac**: A process supervisor — systemd on Linux, launchd on macOS. ```yaml # docker-compose.yml addition voice-worker: build: ./voice-worker environment: - LIVEKIT_URL=ws://livekit:7880 - LIVEKIT_API_KEY=your-key - LIVEKIT_API_SECRET=your-secret - DEEPGRAM_API_KEY=your-key - OPENAI_API_KEY=your-key - ELEVENLABS_API_KEY=your-key restart: unless-stopped depends_on: - livekit ``` The key insight: **the worker is a long-running process, not a request handler.** It maintains a WebSocket connection to LiveKit and responds to room events. Treat it like a daemon, not a web server. ## Outbound calls Your agent can also initiate calls: ```bash # Create a SIP participant in a room (outbound call) lk sip participant create \ --room "morning-briefing" \ --trunk-id "your-trunk-id" \ --number "+15559876543" \ --dtmf "" ``` Or via the API: ```python from livekit import api lk = api.LiveKitAPI( url="http://localhost:7880", api_key="your-key", api_secret="your-secret", ) await lk.sip.create_sip_participant( api.CreateSIPParticipantRequest( room_name="morning-briefing", sip_trunk_id="your-trunk-id", sip_call_to="+15559876543", ) ) ``` Your worker joins the room, the phone rings, the person answers, conversation starts. Use this for scheduled briefings, notifications, or anything where the agent needs to reach out. ## The full Docker Compose ```yaml version: "3.8" services: livekit: image: livekit/livekit-server:latest ports: - "7880:7880" - "7881:7881" - "5060:5060/udp" volumes: - ./livekit.yaml:/etc/livekit.yaml command: ["--config", "/etc/livekit.yaml"] restart: unless-stopped voice-worker: build: ./voice-worker environment: LIVEKIT_URL: ws://livekit:7880 LIVEKIT_API_KEY: ${LIVEKIT_API_KEY} LIVEKIT_API_SECRET: ${LIVEKIT_API_SECRET} DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY} OPENAI_API_KEY: ${OPENAI_API_KEY} ELEVENLABS_API_KEY: ${ELEVENLABS_API_KEY} restart: unless-stopped depends_on: - livekit ``` ## Patterns worth knowing **Caller allowlist.** Check the caller's phone number before letting the agent engage. Reject or route to voicemail for unknown numbers. **Call transcripts.** Capture the full conversation and persist it. Every voice session produces structured data — decisions made, tasks created, questions asked. Feed this back into your agent's memory. **Graceful interruption.** The LiveKit Agents SDK handles barge-in (talking over the agent) natively. But tune the VAD (Voice Activity Detection) sensitivity — too aggressive and it cuts off on background noise, too passive and it feels unresponsive. **Session context.** Load relevant context before the call starts — today's calendar, pending tasks, recent messages. The agent should know what's going on before the human says a word. **Latency budget.** STT (\~200ms) + LLM (\~500-1500ms) + TTS (\~200ms) = \~1-2 seconds minimum. Users tolerate this for the first exchange but it compounds. Keep LLM responses short. Stream TTS (start speaking before the full response is generated). **Voice identity.** Pick a consistent voice. ElevenLabs lets you clone or design voices. The agent should sound the same every time — it's part of the trust relationship. ## Services and costs | Service | Purpose | Rough cost | | -------------------------------------------------------------------------- | -------------------------------- | ------------------- | | [LiveKit](https://livekit.io) | Media server | Free (self-hosted) | | [Telnyx](https://telnyx.com) | SIP trunk + phone number | \~$1/mo + $0.01/min | | [Deepgram](https://deepgram.com) | Speech-to-text | \~$0.0043/min | | [OpenAI](https://platform.openai.com) / [Anthropic](https://anthropic.com) | LLM | Varies by model | | [ElevenLabs](https://elevenlabs.io) | Text-to-speech | \~$0.18/1K chars | | [Cartesia](https://cartesia.ai) | TTS (alternative, lower latency) | \~$0.10/1K chars | Total for a personal voice agent with moderate daily use: roughly $20-50/month on top of your LLM costs. ## Further reading * [LiveKit Agents quickstart](https://docs.livekit.io/agents/quickstart/) * [LiveKit SIP documentation](https://docs.livekit.io/sip/) * [LiveKit Agents SDK reference](https://docs.livekit.io/agents/overview/) * [Deepgram streaming API](https://developers.deepgram.com/docs/streaming) * [ElevenLabs voice design](https://elevenlabs.io/docs) * [My janky experience building this](https://joelclaw.com/voice-agent-deployment-deep-dive) — the honest version