A curated, developer-friendly learning path for building real-time voice AI agents, from your first STT call to scaling production telephony.
English ยท ไธญๆ็ๆฌ
Voice AI has moved from research demos into shipping product in under three years. The modern stack is converging around a clear pattern: a real-time transport layer (WebRTC or telephony), a streaming pipeline of speech-to-text โ LLM โ text-to-speech, and a turn-taking model that decides when the agent should speak. This list is structured to mirror that learning order: start with the foundations, pick a framework, then drill into individual components and production concerns.
Learning resources are tagged ๐ข Beginner, ๐ก Intermediate, or ๐ด Advanced (blogs, podcasts, communities, and conferences in sections 17-20 are intentionally left untagged). Prefer free official docs and vendor-neutral guides; flag where authors have commercial interests.
Read top-to-bottom if you're brand new. The recommended path:
- Foundations โ understand the pipeline and latency budget
- Frameworks โ pick one (LiveKit Agents or Pipecat are the safest open-source bets) and ship a hello-world
- Components (STT, TTS, LLM, VAD, turn detection) โ swap pieces to learn what each layer does
- Transport & telephony โ connect to a real phone number
- Evaluation, production, ethics โ make it safe enough to ship
If you want this material in a tighter, opinionated, production-grade form, I wrote the Voice Agents Handbook: building production voice AI with LiveKit, plus appendices on choosing your stack and the LiveKit ecosystem beyond agents. Available now on Kindle (and in paperback).
The README you're reading collects the field's best free resources. The book is the curated path through them, with the patterns I've used shipping voice agents for trade people, lawyers, and immigration consultants.
Disclosure: I maintain this repo and authored the handbook. Free sample (Introduction + Chapter 1) at handbook.mahimai.ca.
๐ Expand the 21 sections
- Foundational concepts and learning paths
- Frameworks and orchestration platforms
- Speech-to-text (STT / ASR)
- Text-to-speech (TTS)
- LLMs for voice and real-time AI
- Voice activity detection and turn-taking
- Audio enhancement and noise suppression
- WebRTC fundamentals
- Telephony and SIP
- Tutorials and hands-on projects
- GitHub starter repos and awesome lists
- Datasets and benchmarks
- Beginner-accessible research papers
- Evaluation and testing
- Production, deployment, and scaling
- Ethics, safety, and regulation
- Blogs and newsletters
- Podcasts
- Communities
- Conferences and events
- Hackathons and competitions
Start here. These resources establish the mental model of the voice agent pipeline and the latency budget you'll fight for the rest of your career.
6 resources
- ๐ข Voice AI & Voice Agents: An Illustrated Primer: Kwindla Hultman Kramer's free, regularly-updated long-form primer. The de facto textbook for the field.
- ๐ข Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained (LiveKit): Visual walkthrough of streaming patterns, turn detection, and where latency accumulates.
- ๐ข Everything You Need to Know About Voice AI Agents (Deepgram): End-to-end primer covering feature extraction, ASR, LLM reasoning, and synthesis.
- ๐ข AI Voice Agents (LiveKit Docs): The canonical "what is a voice agent" reference, covering the Agents framework, sessions, and the STT-LLM-TTS pipeline vs realtime model split.
- ๐ข Core Latency in AI Voice Agents (Twilio): Visual explanation of end-of-turn detection, silence thresholds, and smart endpointing.
- ๐ก How Intelligent Turn Detection Solves the Biggest Challenge in Voice Agents (AssemblyAI): Endpointing is the most underestimated problem; this is the clearest deep-dive.
The frameworks below all let you wire STT, an LLM, and TTS together. For open-source production work, LiveKit Agents and Pipecat are the two safest bets; for managed dashboards, Vapi, Retell, and Bland win on time-to-first-call.
| Pick | Type | Best for |
|---|---|---|
| LiveKit Agents | Open source | Production, WebRTC-native |
| Pipecat | Open source | Vendor-neutral pipelines |
| Vapi / Retell / Bland | Managed | Fastest first call |
| OpenAI Realtime / GPT-Live / Gemini Live | Realtime API | Speech-to-speech |
20 resources
- ๐ข LiveKit Agents: Voice AI Quickstart: Working assistant in <10 min via Python or TypeScript, runs on top of WebRTC.
- ๐ข Pipecat: Quickstart: Scaffolds a Deepgram + OpenAI + Cartesia pipeline via the Pipecat CLI (
uv tool install "pipecat-ai[cli]", thenpipecat init quickstart); talk to it in the browser in ~5 minutes. - ๐ด Ultravox (fixie-ai/ultravox): Open-weight speech LLM (a Whisper encoder feeding an LLM backbone; v0.7 runs on GLM-4.6, earlier releases on Llama, Gemma, and Qwen) that skips the separate ASR stage.
- ๐ก qwen-audio-agent: Full-duplex voice runtime that drives coding agents (OpenCode, Claude Code, Codex, and other ACP backends) with barge-in, parallel background tasks, and a local wake word.
- ๐ก TEN Framework: Open-source framework for real-time multimodal (voice, video, text) agents with pluggable ASR, LLM, and TTS extensions; licensed Apache 2.0 with additional restrictions and backed by Agora.
- ๐ข Vapi: Quickstart: Dashboard-first; ship an agent on a free US phone number in under 5 minutes.
- ๐ข Retell AI: Introduction & Quickstart: Phone-agent platform with $10 free credit on signup.
- ๐ข Bland AI: Send Your First Phone Call: Minimal API tutorial for placing your first AI phone call.
- ๐ข ElevenLabs Agents: Quickstart: Build and embed a voice agent widget on any website in 5 minutes (formerly branded "Conversational AI," now ElevenAgents).
- ๐ก OpenAI Realtime API: Guide: Official guide to
gpt-realtime-2.1andgpt-realtime-2.1-mini(GA; configurable reasoning) over WebRTC, WebSockets, or SIP. - ๐ก OpenAI GPT-Live: Guide: Full-duplex voice model (
gpt-live-1) that listens and speaks at once, decides when to yield or interrupt, and hands reasoning and tool calls to a backend text model; over WebRTC, WebSockets, or SIP. - ๐ก Google Gemini Live API: Overview: Low-latency, bidirectional voice + vision agents with barge-in and tool use, on
gemini-3.8-live(plus an Extended Thinking variant). - ๐ก Twilio ConversationRelay: WebSocket bridge that handles STT/TTS so you focus on LLM logic; works with any LLM.
- ๐ก ThunderPhone Realtime API: OpenAI-Realtime-compatible speech-to-speech WebSocket for phone agents (47 languages, telephony support), with official Pipecat and LiveKit Agents plugins.
- ๐ก Eleven Realtime (ElevenLabs): OpenAI-Realtime-compatible API (October 2026) with
eleven_realtime_v1_miniandeleven_realtime_v1_max, a cascade of distinct models behind one WebSocket; switch an OpenAI client by changing the endpoint and key. - ๐ก xAI Grok Voice 3.0: Speech-to-speech model (
grok-voice-3.0, October 2026) over WebSocket or SIP with server VAD or manual turns, uninterruptible forced messages, and 8 kHz G.711;grok-voice-2.0is deprecated. - ๐ก PolyAI Dialog-RSN-1: Audio-native reasoning model for calls behind an OpenAI-Realtime-compatible WebSocket; it returns speech, or text only so you keep your own TTS voice.
- ๐ก Vapi vs Pipecat vs LiveKit (AssemblyAI): Architecture-focused comparison of pipeline control and transport choices.
- ๐ข 12 Voice Agent Platforms Compared (Softcery): Broad 2026 market map with use-case recommendations; note the commercial author (a development agency).
- ๐ก Best Voice Agent Stack (Hamming AI): Buy-vs-build framework with concrete cost, latency, and time-to-launch numbers.
Pick one streaming STT and learn it deeply before shopping around. Deepgram, AssemblyAI, and Whisper-derivatives cover most use cases. (All-in-one ASR + end-of-turn models like Deepgram Flux are covered under turn-taking.)
| Pick | Type | Best for |
|---|---|---|
| Deepgram Nova-3 | Commercial | General-purpose, 36+ languages |
| AssemblyAI Universal-3.5 Pro | Commercial | Accuracy, diarization |
| Speechmatics Agent STT | Commercial | Agent turns: names, numbers |
| Soniox | Commercial | Multilingual + built-in translation |
| faster-whisper | Open source | Self-hosted Whisper |
| NVIDIA Parakeet (NeMo) | Open source | Top-of-leaderboard accuracy |
26 resources
- ๐ข Deepgram Nova-3: STT benchmarks: Primer on WER, latency, and cost alongside Deepgram's product reference; Nova-3 spans 36+ languages with multilingual code-switching.
- ๐ก AssemblyAI Universal-3.5 Pro Streaming: Streaming STT walkthrough that doubles as a function-calling tutorial; Universal-3.5 Pro (July 2026) is now the flagship for streaming and async, with code-switching across 18 languages and earlier conversation turns accepted as context mid-call.
- ๐ข OpenAI Speech-to-Text (gpt-transcribe / gpt-live-transcribe): Easiest cloud STT if you already use OpenAI;
gpt-transcribe($0.0045/min) and the streaminggpt-live-transcribe(July 2026) succeed gpt-4o-transcribe. - ๐ข Cartesia Ink 2: GA streaming STT with built-in eager turn detection and noise robustness, paired with Sonic TTS for a single-vendor low-latency stack.
- ๐ข Soniox Speech-to-Text: One model spanning 60+ languages with real-time WebSocket streaming and async APIs, speaker diarization, language identification, endpoint detection, and built-in real-time speech translation (one-way or two-way).
- ๐ก Speechmatics Melia: Single-pass multilingual STT with native code-switching across 55+ languages (batch, production preview).
- ๐ก Speechmatics Agent STT: Streaming STT for voice agents (September 2026) on the Linden model, tuned for names, numbers, negations, and one-word confirmations; 369 ms median finalization in the open Pipecat benchmark.
- ๐ก Gladia Solaria-3: STT tuned for noisy, multi-speaker European business audio (9.6% WER on English production calls).
- ๐ข Gradium STT: Streaming STT with built-in semantic VAD; step messages every 80 ms carry end-of-turn probabilities so agents can decide when a speaker has finished.
- ๐ก Meta Muse Voice Transcribe: One streaming model for ASR, diarization (20+ speakers), and endpointing across 25 languages with code-switching (September 2026); the most accurate model in the Pipecat streaming benchmark, though with slower P95 finalization.
- ๐ข Gemini 3.5 Transcribe: Google's dedicated STT (August 2026) with a batch model and a streaming Live API model (
gemini-3.5-transcribe-live, preview) across 85+ languages. - ๐ข Mistral Voxtral Transcribe 3: Batch Small and Mini models across 50+ languages from $0.0035/min, plus the streaming
voxtral-mini-transcribe-realtime-3-0with sub-500 ms latency (vendor figure) at $0.008/min (October 2026).
- ๐ข openai/whisper: The original repo and the de facto starting point for any DIY ASR project.
- ๐ก SYSTRAN/faster-whisper: CTranslate2 reimplementation up to 4ร faster with INT8; recommended for self-hosted Whisper.
- ๐ด NVIDIA NeMo (Parakeet / Canary): Top-of-leaderboard open ASR models with streaming inference recipes.
- ๐ก FunASR: Industrial open-source ASR toolkit with Paraformer, SenseVoice, VAD, punctuation, diarization, streaming services, and an OpenAI-compatible API for self-hosted voice agents.
- ๐ก SenseVoice: Multilingual speech understanding model for ASR, language identification, emotion recognition, and audio event detection, with FunASR integration and ONNX/libtorch export examples.
- ๐ก Moonshine: On-device voice toolkit (STT, TTS, and an agent loop) with MIT models from 1 MB up to beyond Whisper Large V3 accuracy, running in Python, WASM, iOS, Android, and on Raspberry Pi.
- ๐ด NVIDIA Nemotron 3.5 ASR Streaming 0.6B: Open-weights cache-aware FastConformer streaming ASR across 40 locales, with runtime-configurable latency (80 ms to 1.1 s).
- ๐ก Qwen3-ASR: Open-weight ASR from the Qwen team covering 52 languages with streaming inference; a strong self-hosted multilingual option.
- ๐ข Open ASR Leaderboard (HuggingFace): Community leaderboard with English short-form, long-form, multilingual, and private held-out tracks (WER plus RTFx): your reference for open-source picks.
- ๐ข Artificial Analysis: Speech-to-Text: Independent leaderboard ranking 50+ STT models (batch and streaming) by WER, speed, and cost.
- ๐ก pipecat-ai/stt-benchmark: Open streaming STT benchmark scoring 23 models on 1,000 real voice-agent utterances by semantic WER and time to final segment (median, P95, P99), with code and data to rerun it; note the commercial author.
- ๐ก Best Speech-to-Text Providers in 2026 (Coval): Independent benchmark across 14 providers (WER, latency, end-of-turn, cost), with guidance on testing against your own traffic.
- ๐ข Best Speech-to-Text APIs in 2026 (Deepgram): Provider comparison guide; note the commercial author.
- ๐ก Streaming vs Batch ASR (Arun Baby): Engineer-friendly explainer of RNN-T and Conformer streaming architectures.
Latency, not raw quality, is what kills voice agents: prioritize providers offering true streaming with first-byte under 200 ms.
| Pick | Type | Best for |
|---|---|---|
| ElevenLabs | Commercial | Quality, voice cloning |
| Cartesia Sonic | Commercial | Lowest latency for agents |
| Deepgram Aura-2 | Commercial | Pairs with Deepgram STT |
| Kokoro | Open source | Tiny, CPU-friendly |
| Chatterbox | Open source | Cloning + emotion control |
21 resources
- ๐ข ElevenLabs Docs: Industry-leading quality, voice cloning, and Agents platform in one SDK.
- ๐ข Cartesia Sonic Quickstart: Sonic 3.6 (44 languages, dated snapshots, native turn detection), sub-90 ms first-byte latency, designed specifically for voice agents.
- ๐ข Deepgram Aura-2: Low-latency streaming TTS (Aura-2) that pairs cleanly with Deepgram STT.
- ๐ข OpenAI TTS (gpt-4o-mini-tts): Easiest plug-in TTS for the OpenAI stack, but
gpt-4o-mini-tts,tts-1, andtts-1-hdshut down on January 6, 2027; OpenAI points new voice work togpt-realtime-2.1-mini. - ๐ข Soniox Text-to-Speech: Low-latency streaming TTS over WebSocket with multilingual voices; pairs with Soniox STT and translation.
- ๐ข Gradium TTS: Streaming WebSocket TTS with 158 ms P50 time to first audio and instant voice cloning in five languages.
- ๐ข Gandr TTS: Streaming TTS over SSE or WebSocket with 116 ms p50 time to first audio (vendor figure), six multilingual voices across 23 languages, and watermarked output; ships LiveKit Agents and Pipecat integrations.
- ๐ข Speechify Text-to-Speech API (Simba 3.5): Streaming TTS over HTTP;
simba-3.5covers 48 languages under one model ID, Simba 3.2 and 3.0 add SSML and emotion control, and Simba 3.2 had a 106 ms median time to first audio on Coval's independent benchmark (September 2026). - ๐ข Mistral Voxtral TTS 3: Voxtral Mini TTS 3 (October 2026) with voice cloning across 35+ languages at $13 per 1M characters; Mistral's voice agent SDK, built on LiveKit, pairs it with Voxtral realtime STT.
- ๐ข Artificial Analysis: TTS leaderboard: ELO, price, and speed comparison covering Cartesia, ElevenLabs, Google, Inworld, Rime, Hume, and open-weights models.
- ๐ก Best Text-to-Speech Providers in 2026 (Coval): Independent head-to-head of 14 TTS providers on latency, naturalness, and cost; note the commercial author.
- ๐ก Chatterbox (resemble-ai/chatterbox): Resemble AI's MIT-licensed TTS (vendor-reported to beat ElevenLabs in blind preference tests); ~5 s zero-shot voice cloning, emotion-exaggeration control, and a built-in PerTh watermark. Turbo (350M) hits sub-150 ms first audio, Nano (110M) runs 3ร real time on CPU, and Multilingual (V3, 0.5B) covers 23+ languages.
- ๐ด Fish Audio Drama 3: Open-weight 35B MoE TTS (about 3B active) across 83 languages (October 2026), with 88.9 ms median first audio on one H200 (vendor figure); its community license requires a paid license above $500K annual revenue.
- ๐ข Kokoro 82M (kokoro-onnx): Maintained ONNX runtime for the tiny Apache-licensed Kokoro model that tops community ELO arenas; runs on CPU (the upstream weights have not changed since 2025).
- ๐ข Piper (OHF-Voice/piper1-gpl): Fast local neural TTS optimized for Raspberry Pi; perfect for offline projects.
- ๐ก Coqui TTS (idiap fork): Maintained fork of Coqui-TTS / XTTS v2; still battle-tested, though Chatterbox now leads on zero-shot cloning quality.
- ๐ก Orpheus-TTS: Llama-3B-based emotive TTS with ~200 ms streaming and emotion tags.
- ๐ด Sesame CSM (sesame/csm-1b): Conversational, context-aware multi-speaker TTS using a Llama backbone with the Mimi codec; runs natively in HF Transformers.
- ๐ก Qwen3-TTS: Apache-2.0 open-weight TTS from the Qwen team with streaming output across 10 languages.
- ๐ก Streaming TTS for Low-Latency Agents (Picovoice): Clear taxonomy of single, output-streaming, and dual-streaming TTS.
- ๐ข Ethics of Voice Cloning & Deepfakes (Deepgram): Vendor-neutral discussion of misuse, regulation, and developer responsibility.
A voice agent's perceived intelligence is bounded by how fast the LLM streams its first token. Sub-300 ms TTFT changes the conversation feel entirely.
15 resources
- ๐ข Groq: LPU-based inference cloud; among the lowest time to first token on mainstream open models in 2026 benchmarks.
- ๐ข Cerebras Inference: Wafer-scale chip inference with very high throughput on Llama models.
- ๐ข SambaCloud (SambaNova): Reconfigurable Dataflow inference; stable throughput at low latency.
- ๐ด PhoneLLM Alpha 1 (Pipecat): Open-weight (BSD-2) fine-tune of Nemotron 3 Nano 30B-A3B (3.5B active parameters) built for phone agents, with P95 time to first token under 100 ms on one B200; note the commercial author.
- ๐ก OpenAI Realtime API guide: Flagship S2S product with WebRTC/WebSocket transport (
gpt-realtime-2.1, GA). - ๐ก Google Gemini Live: Real-time multimodal voice/video with barge-in and broad language support, on
gemini-3.8-live. - ๐ด Moshi (kyutai-labs): Open full-duplex speech-text foundation model (~200 ms, Mimi codec). Kyutai's broader stack now includes Unmute (cascaded STT+LLM+TTS with tool use), Kyutai STT/TTS, and Hibiki (streaming translation).
- ๐ด NVIDIA NemotronLabs VoiceChat 11B: Open-weight English full-duplex speech model (August 2026) with tool calling and about 450 ms turn-taking (vendor figure); weights under the OpenMDW 1.1 license, inference code in NeMo Speech.
- ๐ก Speech-to-Speech Models in 2026: Three Architectural Bets (Krzysztof Sopyla): Vendor-neutral comparison of full-duplex (Moshi), near-duplex multimodal (Qwen-Omni), and cascade approaches, with FullDuplexBench numbers and tradeoffs.
- ๐ข Artificial Analysis: Speech-to-Speech: Independent leaderboard for speech-to-speech models on reasoning (Big Bench Audio), conversational dynamics (Full-Duplex-Bench), latency, and price, plus a blind arena.
- ๐ข OpenAI Voice Agents Guide: Compares GPT-Live (full duplex), Realtime sessions, and chained pipelines, with prompt and tool best practices.
- ๐ก ElevenLabs Voice Agent Prompting Guide: Production-grade prompt structure tuned for voice; vendor-neutral lessons.
- ๐ข Voice AI Prompt Engineering Guide (VoiceInfra): Explains why voice prompts must be 60โ70% shorter than chat prompts, with templates.
- ๐ก Tool Definition and Use for Voice Agents (LiveKit Docs): Defining
@function_tooltools and raw-schema tools inside a voice agent. - ๐ก LLM Benchmarks for Voice Agents (Pipecat): 46 model configurations from 7 providers run through scripted 30-turn conversations, scoring tool use, instruction following, and grounding inside a voice latency budget; note the commercial author.
Pure VAD is no longer enough: modern agents combine acoustic VAD with a small semantic model that predicts end-of-utterance from words and prosody.
13 resources
- ๐ข Silero VAD: MIT-licensed pre-trained VAD; <1 ms per chunk on CPU. The de facto VAD inside LiveKit and Pipecat.
- ๐ก LiveKit Turn Detector v1.0: Audio-native end-of-turn model (fused semantic + acoustic, no transcript) across 14 languages; now the LiveKit default.
- ๐ก Deepgram Flux: All-in-one conversational STT with built-in end-of-turn detection (English median EOT <300 ms; Flux Multilingual adds 10 languages with mid-call switching), integrated with Deepgram's Voice Agent API; collapses STT and turn detection into a single model.
- ๐ก Pipecat Smart Turn v3.2: Open BSD-2 audio-native turn detection covering 23 languages; the 8 MB int8 CPU build runs in about 10 ms on fast CPUs, with better handling of background noise and short replies.
- ๐ก pipecat-ai/smart-turn: Repo with model code, training scripts, and integration examples (~8M params, Whisper-Tiny base).
- ๐ก Krisp Turn-Taking v2 (VIVA SDK): Commercial audio-only turn-taking model with interruption prediction that separates barge-ins from backchannels; works with any STT/LLM/TTS stack.
- ๐ข The Complete Guide to AI Turn-Taking (Tavus): Reader-friendly overview of why pure VAD fails in real conversations.
- ๐ก Tackling Turn Detection in Voice AI (Notch): Engineer-first walkthrough combining VAD probability, volume, and TTS markers.
- ๐ก What Is Endpointing in Voice AI? (Cekura): Explainer on the three-signal endpointing stack with a testing angle; note the commercial author.
- ๐ก Evaluating End-of-Turn Detection Models (Deepgram): Methodology plus a head-to-head of Flux, Pipecat Smart Turn, and LiveKit EOU; note the commercial author.
- ๐ก eot-bench (LiveKit): Open, reproducible end-of-turn benchmark that scores models at real pauses under latency and false-cutoff budgets, with an Apache-2.0 dataset of real human-to-agent turns in 14 languages; note the commercial author.
- ๐ก Semantic VAD (Gradium): Explainer on multi-horizon end-of-utterance prediction, with streaming inactivity probabilities every 80 ms and guidance on tuning the latency vs interruption tradeoff; note the commercial author.
- ๐ข ai-coustics VAD: VAD bundled with real-time speech enhancement, noise suppression, and voice isolation in a single audio preprocessing SDK; useful when you want cleanup and turn-taking signals from the same component.
The audio reaching your VAD and STT is often noisy, reverberant, or mixed with background voices. Cleaning the signal before the rest of the pipeline is frequently the difference between an agent that ships and one that frustrates users in real-world conditions (cars, cafรฉs, call centres). In 2026 every major voice-AI vendor ships a deep-learning suppressor on top of WebRTC's classic noise-suppression chain.
6 resources
- ๐ข ai-coustics: Real-time speech enhancement SDK covering noise cancellation, voice isolation, and VAD; on-device and cloud deployment. See the docs and developer platform.
- ๐ข Krisp VIVA SDK: Commercial-grade real-time noise and background-voice cancellation, with VAD and turn-taking in the same SDK; the de facto standard for voice comms. LiveKit's background voice cancellation and Pipecat Cloud both build on Krisp.
- ๐ก DPDFNet (ceva-ip/DPDFNet): Apache-2.0 real-time speech enhancement that extends DeepFilterNet2 with dual-path RNN blocks; 2.3M to 3.6M-parameter models for 8, 16, and 48 kHz audio with ONNX and TFLite exports; note the commercial author.
- ๐ก GTCRN (Xiaobin-Rong/gtcrn): Ultra-light speech enhancement model (48.2K parameters, 33 MMACs/s, ICASSP 2024, MIT); a tiny, well-understood baseline for constrained devices.
- ๐ข Koala Noise Suppression (Picovoice): On-device, cross-platform voice isolation with self-serve access (browser, mobile, desktop, Raspberry Pi).
- ๐ก Noise Suppression Guide 2026 (Picovoice): Algorithms, intelligibility metrics (SII / STI / STOI), and implementation tradeoffs; note the commercial author.
WebRTC is the default transport for voice agents that don't run over the phone network. Understanding ICE, STUN, TURN, and SFU architecture is non-negotiable for production work.
10 resources
- ๐ข MDN WebRTC API: Authoritative free reference for
RTCPeerConnection,getUserMedia, and signaling. - ๐ข MDN: Introduction to WebRTC Protocols: Beginner-friendly explanation of ICE, STUN, TURN, and SDP.
- ๐ข WebRTC.org Getting Started: Official Google-maintained intro, splitting WebRTC into media-capture and connectivity.
- ๐ข GetStream: WebRTC for the Brave: Free multi-module tutorial covering networking basics through advanced topics.
- ๐ก Why WebRTC Beats WebSockets for Voice AI (LiveKit): 2025 explainer aimed at AI builders, comparing transports in plain English.
- ๐ข Daily Docs: Intro to Video Architecture (P2P vs SFU): One of the clearest beginner write-ups of P2P vs SFU.
- ๐ก P2P, SFU, MCU, Hybrid: WebRTC Architecture Guide (Forasoft): 2026 breakdown of the four architectures with current OSS tooling (mediasoup, Janus, Jitsi); note the commercial author (a development agency).
- ๐ข Agora: How WebRTC Works: Side-by-side WebRTC vs WebSockets walkthrough with signaling diagrams.
- ๐ด How OpenAI Delivers Low-Latency Voice AI at Scale: OpenAI's production WebRTC design, with a stateless edge relay and a stateful transceiver that owns ICE, DTLS, and SRTP so media runs in Kubernetes behind a small fixed UDP surface; note the commercial author.
- ๐ด OpenAI's WebRTC Problem (moq.dev): Counterpoint from the Media over QUIC project arguing that WebRTC's multi-round-trip setup and aggressive packet dropping fit voice AI poorly, and sketching a MoQ alternative.
The phone network has its own physics. Once you know which SIP trunk provider to point at LiveKit or Pipecat, you can ship.
| Pick | Type | Best for |
|---|---|---|
| Twilio | Commercial | Default, largest ecosystem |
| Telnyx / Plivo | Commercial | Cost-effective SIP trunks |
| SignalWire | Commercial | FreeSWITCH-based, programmable |
| LiveKit SIP / Pipecat | Framework | Wire trunks into your agent |
11 resources
- ๐ข Twilio Programmable Voice: TwiML, Voice API, and PSTN connectivity in one hub; the default starting point.
- ๐ข Twilio: Voice AI Assistant with OpenAI Realtime + Python: Step-by-step junior-friendly tutorial wiring Twilio Media Streams to an LLM.
- ๐ข Twilio SIP Quickstart: Clearest beginner explainer of SIP basics, SIP Domains, and softphone setup.
- ๐ข Telnyx Voice API: Strong Twilio alternative with WebSocket media streaming and AI Assistant tooling.
- ๐ข Telnyx: How to Set Up a SIP Trunk: Friendly walkthrough of SIP trunking architecture, codecs, and authentication.
- ๐ข Plivo Voice API Documentation: XML call control and audio-streaming integrations for AI agents.
- ๐ก SignalWire Voice Docs: Built on FreeSWITCH; SWML, TwiML-compatible API, and an AI Agents SDK.
- ๐ข LiveKit SIP Primer: Best diagram of how a call flows from PSTN โ trunk โ SIP service โ agent.
- ๐ก LiveKit SIP Trunk Setup: Practical guide for wiring Twilio/Telnyx/Plivo/Wavix/Sinch trunks into LiveKit.
- ๐ก OpenAI Realtime API with SIP: Point a SIP trunk at OpenAI's SIP endpoint and accept each call from a webhook, configuring the session per call with no media bridge server; an EU endpoint is available.
- ๐ก Pipecat Telephony Overview: Differences between WebSocket-based telephony and SIP-based call control.
Pick one tutorial and finish it before starting another. Voice AI is unforgiving of half-built pipelines.
8 resources
- ๐ข Build Your First AI Voice Agent in Python (LiveKit): End-to-end Python tutorial covering streaming, latency, and deployment.
- ๐ก How to Build a Real-Time Voice Agent with Pipecat (AssemblyAI): Production-oriented walkthrough including local testing and Pipecat Cloud deployment.
- ๐ก Build a Voice Agent with LiveKit (AssemblyAI): End-to-end walkthrough wiring LiveKit Agents + AssemblyAI Universal-3 Pro + Cartesia, run locally then on the Agents Playground.
- ๐ข Deepgram: Build a Voice AI Agent: Step-by-step guide wiring Deepgram STT, GPT, and Aura TTS.
- ๐ก Build a Voice Assistant with Twilio ConversationRelay + LiteLLM: Provider-agnostic tutorial supporting OpenAI, Anthropic, or DeepSeek.
- ๐ข freeCodeCamp: Build Advanced AI Agents (LiveKit, Exa, LangChain): Free 3-part video course covering interactive voice agents end-to-end.
- ๐ก freeCodeCamp: Build a Voice AI Agent with Open-Source Tools: Hands-on local stack covering open-source STT, a local LLM, and system TTS, plus the cascaded vs end-to-end tradeoff.
- ๐ข DeepLearning.AI: Voice for AI Agents and Applications: Free short course (June 2026) on three voice integration patterns: embedded, layered on a text agent, and voice as a callable tool.
Clone these instead of writing boilerplate from scratch.
7 resources
- ๐ขโ๐ด livekit/agents: The flagship open-source Python/Node framework for production voice agents (tip: pair it with the LiveKit Docs MCP server and Agent Skill for AI-assisted builds).
- ๐ขโ๐ด pipecat-ai/pipecat: Vendor-neutral framework with 80+ STT/LLM/TTS service integrations.
- ๐ข livekit-examples/agent-starter-python: Production-ready starter with Dockerfile, CI-run conversation simulations, a multilingual turn detector, and core plugins.
- ๐ข livekit-examples (org): Official collection of LiveKit Python/React/Swift/Android starters.
- ๐ขโ๐ก pipecat-ai/pipecat-examples: Sample apps for push-to-talk, websocket, telephony, and multimodal use cases.
- ๐ข elevenlabs/elevenlabs-examples: Runnable Next.js and Python examples for TTS, STT, and real-time agents.
- ๐ข wildminder/awesome-ai-voice: Actively maintained 2026 list of open-source TTS, voice-cloning, and audio/music-generation models.
You'll rarely train from scratch, but knowing which dataset a model was trained on explains its accents, languages, and failure modes.
8 resources
- ๐ข LibriSpeech ASR Corpus: ~1,000 hours of English audiobooks; nearly every ASR paper benchmarks against it.
- ๐ข Mozilla Common Voice: Crowdsourced multilingual dataset (295 languages in v27.0, CC0); the easiest legal way to fine-tune ASR.
- ๐ข Common Voice on Mozilla Data Collective: The official home of Common Voice downloads since October 2025 (Scripted Speech 27.0 and Spontaneous Speech 5.0), via browser, API, or a Python SDK; free account required.
- ๐ข LJSpeech Dataset: ~24 hours of single-speaker English audio; baseline corpus for Tacotron 2 and VITS.
- ๐ก VCTK Corpus: ~110 English speakers with diverse accents; widely used for multi-speaker TTS.
- ๐ก VoxCeleb (Oxford VGG): Million-utterance "in the wild" dataset for speaker identification and verification.
- ๐ก TurnBench: Hand-labeled benchmark for end-of-turn and interruption detection (August 2026), with a 30 h two-channel test set (154 conversations, 106 speakers) and a 104 h training set, scored across 14 systems; the data is non-commercial.
- ๐ข Coval Voice AI Benchmarks: Continuously updated, reproducible STT, TTS, and speech-to-speech latency and accuracy benchmarks on call-like audio across about 40 providers, with open code; note the commercial author.
These are the landmark papers behind the models you'll actually use. Read the Whisper and Common Voice papers first: they're unusually approachable.
10 resources
- ๐ก Whisper: Robust Speech Recognition via Large-Scale Weak Supervision (2022): Behind the most popular open ASR model; unusually clear prose for an ML paper.
- ๐ข HuggingFace Whisper fine-tuning blog (companion): Hands-on walkthrough that lets you "feel" the Whisper paper in code.
- ๐ก VITS: Conditional VAE with Adversarial Learning for End-to-End TTS (2021): The single-stage TTS model behind many open-source voice cloners.
- ๐ก Tacotron 2: Natural TTS Synthesis (2017): Landmark seq2seq + WaveNet-vocoder paper that made neural TTS sound natural.
- ๐ก Conformer: Convolution-augmented Transformer for ASR (2020): The architecture inside NVIDIA Parakeet, Canary, and many leaderboard models.
- ๐ก wav2vec 2.0: Self-Supervised Learning of Speech Representations (2020): Showed that pretraining on unlabeled audio drastically reduces labeled-data needs.
- ๐ข Common Voice: A Massively-Multilingual Speech Corpus (2020): Short, accessible paper describing how Common Voice is built and validated.
- ๐ด Moshi: A Speech-Text Foundation Model for Real-Time Dialogue (2024): The first real-time full-duplex spoken LLM; introduces the Mimi codec and the "Inner Monologue" method (time-aligned text before audio tokens).
- ๐ก Open ASR Leaderboard preprint (2025): Reproducible benchmark of 60+ ASR models across 11 datasets; the modern landscape map.
- ๐ก Full-Duplex-Bench: Evaluating Full-Duplex Spoken Dialogue Models on Turn-Taking (2025): A reproducible benchmark for interruption handling and turn-taking in speech-to-speech models.
You can't ship what you can't measure. Voice-agent evaluation is fundamentally probabilistic: a single transcript can pass and fail across runs, so simulation and statistics matter more than fixed test cases.
13 resources
- ๐ข Coval: Voice AI Testing Platform: Defines the core voice-agent metrics: TTFB, WER, resolution rate, simulated accents, and interruptions.
- ๐ข Coval: How to Evaluate Voice Agents (Practical Guide): One of the most cited 2025 guides on probabilistic vs deterministic evaluation.
- ๐ข Cekura: Metrics Overview: Predefined metrics, instruction-following checks, and simulation framework.
- ๐ก Cekura: Performance Testing for Voice Agents: Practical 2025 guide on multi-turn simulation and edge-case generation.
- ๐ก Hamming AI: Production-focused QA platform with simulation, load testing, and 50+ metrics.
- ๐ก Hamming: Voice Agent Evaluation Metrics Guide: Reference of latency percentiles, WER, MOS-style quality, and task completion with formulas.
- ๐ก LiveKit: Understand and Improve Agent Latency: Per-turn latency metrics (e2e, LLM TTFT, TTS TTFB) and where to optimize.
- ๐ข Twilio: How Do You Know if Your Voice AI Agents Are Working?: 2025 guide arguing for business-outcome metrics over raw WER/latency; note the commercial author.
- ๐ก Future AGI: Open-source platform to simulate, evaluate, trace, guardrail, and optimize voice and AI agent apps in one feedback loop, with persona-driven simulation (through its Simulate SDK) and 50+ eval metrics.
- ๐ก Roark: Voice-AI QA and observability (YC W25) that turns failed production calls into replayable regression tests.
- ๐ก Cekura for Agents (MCP server): MCP server that lets coding agents (Claude Code, Cursor, Codex) trigger and schedule voice-agent test runs.
- ๐ก EVA (ServiceNow): Open-source bot-to-bot framework that scores cascaded and speech-to-speech voice agents on accuracy and experience across 213 enterprise scenarios; none of the 12 systems tested clears 0.5 on both; note the commercial author.
- ๐ก LiveKit Agent Simulations: Runs an agent against LLM-driven simulated users that follow a scenario, in text mode on every commit or audio mode before release, and returns a pass or fail verdict with the transcript; in beta on LiveKit Cloud, note the commercial author.
Real production voice infrastructure is the hardest unsolved problem in this space. Read these before quoting anyone a per-minute price.
10 resources
- ๐ก LiveKit: Deploy and scale agents on LiveKit Cloud: Real-world write-up on stateful load balancing, autoscaling, and warm pools.
- ๐ก LiveKit: Why You Shouldn't Build Voice Agents Directly on Model APIs: Honest breakdown of what raw model APIs don't give you.
- ๐ก Latent Space: OpenAI Realtime API: The Missing Manual (2024): Field-tested guide from Pipecat's creator on Realtime API production realities; written for the 2024 preview, but its lessons on state, interruptions, and cost still apply.
- ๐ก TWIML: Building Voice AI Agents That Don't Suck (Kwindla Kramer): One-hour discussion on real production architecture and turn-taking.
- ๐ก AWS: Voice Agents with Pipecat and Amazon Bedrock: Full architecture walkthrough including latency optimization and Nova Sonic.
- ๐ข Deepgram: STT API Pricing Breakdown: Vendor-by-vendor per-minute economics: required reading before signing any contract.
- ๐ก Sierra: Shipping and Scaling AI Agents: Case-study on Sonos, SiriusXM, and OluKai voice deployments.
- ๐ก Sierra: Constellation of Models: How a leading CX company composes 15+ models per agent.
- ๐ข LiveKit Agent Observability: Built-in tracing, transcripts, and per-stage latency for LiveKit Cloud.
- ๐ก LiveKit: OpenTelemetry Traces: Exports each agent session's spans (turns, STT/LLM/TTS stages, tool calls) to any OTLP backend, with a Langfuse walkthrough; note the commercial author.
If you're shipping a voice agent in 2026, disclosure and consent are no longer optional. EU AI Act Article 50 has applied since 2 August 2026, and the FCC treats AI voices as artificial under the TCPA.
11 resources
- ๐ข FCC: AI-Generated Voices in Robocalls Illegal (Feb 2024): The landmark TCPA ruling every U.S. voice-agent dev must read.
- ๐ก EU AI Act: Article 50 (Transparency for Deepfakes & AI Interactions): Authoritative text of EU disclosure rules; the obligations have applied since 2 August 2026 (generative systems already on the market before then have until 2 December 2026 for the Article 50(2) machine-readable marking duty).
- ๐ก European Commission: Guidelines on Article 50 Transparency Obligations: Official non-binding guidelines (20 July 2026) on when users must be told they are talking to an AI, how to mark synthetic audio, and how to label deep fakes.
- ๐ก European Commission: Code of Practice on AI-Generated Content: Official EU implementation guidance on watermarking and labelling; the finalized Code was published on 10 June 2026.
- ๐ก Digital Omnibus on AI: Regulation (EU) 2026/1744 (EUR-Lex): In force since 27 July 2026; delays high-risk AI Act obligations to 2027 and 2028 but leaves the Article 50 disclosure rules on their 2 August 2026 schedule.
- ๐ข FTC: Approaches to Address AI-Enabled Voice Cloning: Plain-English summary of the Voice Cloning Challenge winners and Impersonation Rule.
- ๐ข FTC: Proposed Rule on AI Impersonation of Individuals (Feb 2024): Direct source on U.S. impersonation-fraud rules covering AI deepfakes.
- ๐ข Pindrop: Voice Intelligence & Security Report: Industry report documenting the sharp rise in deepfake fraud attempts.
- ๐ข Voice Cloning Ethics (CAMB.AI): Practical overview of consent frameworks, ELVIS Act, and EU AI Act.
- ๐ข What Is SynthID, and Why Is ElevenLabs Using It? (ElevenLabs Docs): ElevenLabs adopts Google DeepMind's inaudible SynthID watermark across generated audio and ships a free public Audio Detector.
- ๐ก NCLC: Top Six TCPA/Robocall Developments 2024/2025: Consumer-protection lens on what's actually being enforced.
Subscribe to two or three to stay current: the field moves quickly.
9 resources
- LiveKit Blog: Engineering deep-dives on WebRTC, agents framework releases, and production patterns.
- Deepgram Learn: Tutorials on STT/TTS, voice agent design, evals, and pipeline architecture.
- Cartesia Blog: State-space TTS models, Sonic releases, and yearly "State of Voice AI" reports.
- ElevenLabs Blog: Product and research announcements with implementation notes.
- Daily.co Blog (Pipecat): Posts from Pipecat's maintainers covering scaling and feature releases.
- Voice AI & Voice Agents: An Illustrated Primer: Free, regularly-updated long-form primer.
- Voice AI Space: Vendor-neutral hub for the voice AI ecosystem: a curated product and tool directory, the Voice AI Newsroom, tutorials and repos, a jobs board, and community meetups.
- Voice AI Newsletter (Krisp): "Future of Voice AI" interview series with founders.
- Voice AI Weekly (Vapi): Weekly Substack rounding up news, products, and tools.
4 resources
- Deepgram AI Minds: Founder and builder interviews across the voice AI ecosystem.
- The Future of Voice AI (Krisp): Weekly founder interviews focused on enterprise voice AI architecture.
- TWIML AI Podcast: voice episodes: Strong technical interviews; the Kwin Kramer episode is a great starting point.
- This Week In Voice (Project Voice): News-roundtable format covering conversational AI.
9 resources
- LiveKit Community Slack: Direct access to maintainers and other agent builders.
- Pipecat Discord: Active community with weekly office hours; invite link from the homepage.
- HuggingFace Discord: #ml-for-audio-and-speech: 200k-member server with strong audio/speech channels.
- Vapi Discord: Builder community for Vapi voice agents; invite from the homepage.
- Retell AI Community: Forum for Retell developers building phone-call voice agents.
- ElevenLabs Discord: Large TTS, voice cloning, and Conversational AI community with daily help threads.
- Deepgram Discord: STT/TTS/Voice Agent API support and build-with-us threads.
- Reddit: r/LocalLLaMA: Active threads on local Whisper/Parakeet, on-device TTS, and end-to-end voice stacks.
- Reddit: r/AI_Agents: General AI-agent community where voice topics surface frequently.
6 resources
- AI Engineer World's Fair: Biggest AI-engineering conference; the Voice track has hosted major launches from ElevenLabs, Vapi, LiveKit, Pipecat, and Cartesia. The 2026 edition ran 29 June - 2 July 2026 at Moscone West, San Francisco; session recordings post free to the AI Engineer YouTube channel.
- AI Engineer YouTube channel: All World's Fair and Summit talks are posted free; the best library of recent voice-AI talks.
- AI Engineer Summit Online 2025: Voice playlist: Curated playlist including voice-track sessions from leading labs.
- AIEWF 2026 Trends (Latent Space): Written recap of the five trends that defined the 2026 World's Fair (29 June - 2 July, San Francisco).
- AGENTIC AI Summit (Modev, formerly VOICE & AI): Long-running Modev voice conference, now broadened to agentic AI; a one-day event on 7 October 2026 at the Carahsoft Conference Center, Reston, Virginia.
- Interspeech 2026: Top academic speech-science conference; intimidating but worth knowing, since most landmark papers debut here. Sydney, Australia, 27 September - 1 October 2026; Interspeech 2027 moves to Sรฃo Paulo, Brazil (29 August - 2 September 2027).
3 resources
- ๐ข ElevenHacks: A season of 11 weekly themed hackathons (spring 2026, $240K+ in prizes); the archive of briefs and winning projects is a good source of voice-agent project ideas.
- ๐ข lablab.ai AI Hackathons: Continuous calendar of short online hackathons frequently sponsored by voice-AI vendors.
- ๐ข Devpost: Voice AI Hackathons: Centralized search for active voice-AI hackathons; the best way to find what's open right now.
- Week 1: Foundations: Read the LiveKit pipeline post and Voice AI Illustrated Primer (sections 1, 8).
- Week 2: First agent: Finish the LiveKit or Pipecat quickstart end-to-end (sections 2, 10).
- Week 3: Components: Swap STT, TTS, and LLM providers; benchmark latency (sections 3, 4, 5).
- Week 4: Turn-taking, audio cleanup & telephony: Add Silero VAD, a turn detector, and a speech-enhancement pass; connect a SIP trunk (sections 6, 7, 9).
- Week 5: Production: Add evaluation, observability, and read the FCC/EU AI Act material (sections 14, 15, 16).
- Ongoing: Subscribe to two newsletters, follow Voice AI Space, and join the Voice AI community on LinkedIn group (sections 17, 18, 19).
Pull requests welcome. Resources must be active in the last 12 months, accessible to developers, and vendor-neutral or clearly labeled when authored by a commercial party. Open an issue to suggest additions or removals. See CONTRIBUTING.md for the full contribution guide.
MIT. Fork it, ship it.