Deploying Voice Agents to Production

Fri, 09 May 2025 00:00:00 +0000

Here are my notes on the session about deploying voice agents to production which is part of the Voice Agents Course.

The course if held by kwindla und swyx, the CTO und an investor of Daily.co, a WebRTC und Voice AI infrastructure provider. Some of their recommendations might be predisposed. I still state them as is as I trust them and because I don’t have enough experience with voice agents in production.

TL;DR
#

Use a voice AI provider for simple, scalable deployment for production.
Use a single VM or your homelab for demos.

Differences between voice agents and traditional web apps
#

are mostly in the transport
persistent connnection (minutes)
bidirectional streaming
stateful sessions

Voice agents in production need
#

A http service for
- API endpoints,
- a website, and
- webhooks,
- spawning bots.
A media transport layer or service:
- WebRTC based for client-to-server (udp), or
- websocket based for server-to-server (tcp).
Bots (udp or tcp, connect to media transport layer)

Bots
#

Are instances of the agents.
Can be written in Python with PipeCat.
Use STT, LLM, and TTS providers, which also are the main cost and latency drivers.
Usually come packaged with small models, eg. for voice activity detection (VAD)
Each spawned bot serves one session and needs allocated resources during the whole session:
- 0,5 vCPU
- 1 GB RAM
- 40kbps for WebRTC audio (in 30-60 kbps range)
- video requires more CPU (eg. 1 vCPU), and bandwidth
Need to be quickly available. Target time-to-first-word:
- 2-3 secs (web),
- 3-5 secs (phone)

Ways to solve the “fast start challenge”
#

percentage-based warm pool
fast startup times (caching, pre-loading)
proactive/predictive scheduling
fallbacks from reactive world (eg. UX based solutions, not just silent fails)

Infra providers
#

need to support tcp and udp
voice ai providers are easiest (Pipecat Cloud, Daily, Vapi, Layercode)
Fly.io (and potentially other container platforms) are good if they support udp (Fly does)
ML focused provides are good for converged bots with larger models included (gpu clouds)
hyperscalers are flexible but complex
BTW: CloudRun does not support udp
demos can run on single VMs or even be served from a home lab
by serving everything converged, time-to-first word can get down to 500ms
otherwise 800-1000 ms is good enough and achievable
proximity to users matters (Daily plans global regions for PipeCat cloud, currently only us-west)
conn between servers can be implemented with WebSockets

What’s next?
#

After the session I have looked at the PipeCat examples and realized it should be easy enough to run a basic voice agent with PipeCat’s SmallWebRTCTransport on a virtual server hosted in Europe and then switch the transport and deploy it to production on PipeCat Cloud.

I will probably try that to see if the latency between US based PipeCat cloud and users in Europe is low enough for a good user experience.

An Overview of the Voice AI Landscape (Session Notes)

Thu, 08 May 2025 00:00:00 +0000

I’m so happy to be part of the Voice Agents Course by Kwindla and swyx. Yesterday, Kwindla kicked it off with an overview of the voice AI landscape. The pace, insights, and questions from the audience were just great.

Here are my personal notes, probably incomplete and maybe not always correct. For a more authoritative overview of the Voice AI landscape, check out their free online book Voice AI & Voice Agents - An Illustrated Primer.

Voice AI has highly valuable use cases with actual real business value.
Benefits:
- Today: lower cost.
- Soon: Better. (peak load response, better answers than most humans can give)
RAG is still important
Some challenges:
- latency
  - measure regularly end-to-end from/to clients,
  - record conversation with mic,
  - visually look at gaps in waveforms,
  - test calls from different regions and cell phone providers
  - aim for 800ms, tough but possible to hit with hosted inference,
  - very optimized/limited deployments can hit 500ms with quality compromises
  - 1000ms is not uncommon (still makes users happy)
- turn detection
  - from VAD to semantic
  - OpenAI shipped a good text mode TDM
  - Gemini flash in audio is ok, but needs to run as parallel flow (“greedily”)
  - LifeKit vs. PipeCat is text vs. audio / end of speech, no audio cues vs. with audio cues
  - both a hard ML challenge and important for experience
- interruption handling
- context management
- function calling, tool use
- sripting, instruction following
standard architecture
- today: 3 models (STT - LLM - STT), easier to achieve robust results with LLMs in text mode
- future probably converged
- 3 models, because
  - LLMs’ text mode is their mode
  - we ride on the edge what the best models can do
  - today, we need to use their best mode
- other use cases, like language learning, can better leverage the benefits of speech-to-speech (1 model)
models used (speed and quality, time-to-first token/byte more important than tokens/s)
- STT:
  - Deepgram
  - Whisper (optimized for streaming, original not built for streaming), down at 400
  - Gladia (for non-english languages)
- LLM:
  - GPT-4o
    - still the workhorse, still more than 4.1
    - big model changes take work and good evals (nobody has good ones),
    - models usually not optimized for voice AI,
    - not yet better results
    - 4o-mini was cheaper but slower and worse for tool/function
    - note from a fellow student: 4.1-mini might be interesting
  - Gemini 2.0 Flash
    - very good, fast, cost efficient
    - best audio model for voice AI, today (gemini in audio input mode for voice-to-voice)
    - multi-lingual input is ok, output not so much (use English)
  - mid-sized open models get better, esp. fine-tuned Llama4 (upcoming OpenPipe finetuning session)
  - additional notes about LLM use in voice AI:
    - reliable function calling/tool use is mostly a question of how to prompt 4o against Gemini
    - llama can get there
    - evals are important as ever
- TTS:
  - PlayAI, Grok
  - OpenAI, Google
  - Cartesia
  - Rime
  - Elevenlabs
- network transport:
  - why network? We don’t get capable enough models on mobile or laptop (yet),
  - hybrid architectures might be relevant, already
  - telephone is a great transport for voice AI, too
    - PSTN is with a phone number (eg. from Twilio)
    - SIP is for interconnectivity with digital telephony infra
  - WebSockets are not good for real time client-to-server audio (and video), but ok for server-to-server audio
  - WebRTC is best, complex, but PipeCat supports it ootb, local for testing, in the cloud offering for production,
- Voice AI building blocks in 2025
  - evals - huge topic
  - hosting and scaling - very different (if you love k8s, your topic)
  - workflow/multi-agent/state machines
  - “perfect” speech
    - LLMs in text mode have passed the turing test
    - not quite reached the point for real-time audio recognition and generation
    - eg. accurately recording email, postal addess
  - human-like turn detection
    - very important for qualitative experience
    - fun and hard ML problem
- what’s next?
  - speech to speech models
  - realtime video
  - programming with voice
  - voice as universal user experience
  - LLM as a judge
orchestration and flows questions
- voice in text out can be done with PipeCat aggregator (no great example, yet)
- accents and voice models: Gladia input, PlayAI output
- tip for Gemini: if you want accents, specify your country.
- how to solve cold-start?
  - Daily solved it for us
  - own infra: Combine optimized startup and warm capacity
- faster responses:
  - greedily inference/speculative before turn-detection
  - helpful but more costly
- multi-speaker:
  - hard, no great solution, so far.
  - training data doesn’t map well to multi-person/agent conversation
  - challenge: know when models should or shouldn’t respond (emit a no-response-token)
- observability tooling (multiple vendors, eg. Coval, will be in Discord and hold sessions)

Even after the first session, I can already say: If you are interested in Voice AI: Take the course!.

Voice Agents on Mitja Martini

Deploying Voice Agents to Production

TL;DR #

Differences between voice agents and traditional web apps #

Voice agents in production need #

Bots #

Ways to solve the “fast start challenge” #

Infra providers #

What’s next? #