Voice AI Latency: Engineering Sub-300ms Conversations
A practical engineering guide to voice AI latency covering endpointing, WebRTC, streaming audio, realtime models, tool calls, barge-in, telephony, latency budgets, p95 monitoring, and production optimization.
Voice AI Latency: Engineering Sub-300ms Conversations
Low-latency voice AI is not created by choosing a fast model and hoping for a natural conversation. It comes from measuring the full audio path, shortening endpoint detection, streaming instead of batching, reducing network hops, controlling tool latency, supporting interruption, and optimizing the slowest percentile rather than one impressive demo.
The direct answer
To engineer fast voice AI, optimize the time between the user finishing a meaningful turn and hearing the first useful audio from the agent. That path includes endpoint detection, transport, model processing, optional tool calls, speech generation, buffering, and playback.
Trilops has optimized a production voice path to below 300 milliseconds in a specific measured configuration. We do not treat that number as a universal end-to-end guarantee. Network conditions, endpointing, telephony, tools, model choice, geography, and conversation state can all move the result. The useful engineering target is a measured latency budget with p50 and p95 behavior, not one best-case screenshot.
One word hides several clocks
What does voice AI latency actually mean?
Teams often report "latency" without defining where the timer starts and stops. That makes comparisons almost meaningless. A voice product can have fast model generation and still feel slow because endpoint detection waits too long. It can also produce a quick first audio chunk while taking much longer to complete a tool-backed answer.
How long until the system decides the user is done?
This is controlled by voice activity detection, semantic turn detection, silence thresholds, audio quality, and conversation behavior.
How long until useful inference begins returning?
This reflects transport, service load, context size, model choice, and whether the architecture is streaming.
How long until the user hears the beginning of the reply?
This is one of the most useful conversational metrics because people notice silence between turns immediately.
How long until the requested operation is actually complete?
A response may start quickly while a booking, eligibility check, lookup, or record update continues behind it.
How quickly does the agent stop when the user speaks?
Slow cancellation makes an otherwise fast agent feel robotic because it keeps talking over the caller.
What happens to the slowest real conversations?
p95 and p99 behavior expose network variability, tool spikes, queueing, retries, and outlier payloads hidden by averages.
Never publish a latency number without defining the start event, stop event, percentile, geography, transport, and whether a tool call was involved.
Conversation is turn taking, not benchmark theater
Why does sub-300ms voice latency matter?
Human conversation depends on timing. Large pauses make people repeat themselves, assume the system failed, or begin another thought just as the agent starts speaking. Very aggressive endpointing creates the opposite failure: the system cuts users off before they finish.
The practical goal is therefore not "lowest milliseconds at any cost." It is fast, stable turn taking with correct interruption behavior and enough context to avoid premature responses. The optimal threshold differs by use case. A short appointment confirmation can be more aggressively tuned than a complex clinical intake answer where users pause naturally while recalling information.
Silence feels longer than compute
Users do not care which service is waiting. They experience one gap between their speech and the agent's reply.
Fast but wrong is worse
A response that starts instantly but mishears identity, date, medication, or intent creates rework and erodes trust.
Latency changes speaking behavior
When the agent is slow, callers repeat themselves, add filler, or interrupt, which increases downstream recognition and state complexity.
Long turns increase call cost
Telephony minutes, realtime model usage, and agent occupancy can all rise when every turn contains avoidable waiting.
OpenAI's current Realtime documentation describes realtime sessions as the appropriate path for live audio that needs low latency, and supports realtime audio over WebRTC, WebSocket, and SIP. For browser and mobile clients, its WebRTC guide recommends WebRTC as the more robust client-side connection option.
Architecture determines how many serial waits you create
Which voice AI architecture produces the lowest latency?
There are two common patterns: a native speech-to-speech realtime path and a chained speech-to-text, language-model, text-to-speech path. Neither is universally better. The right choice depends on latency, transcript requirements, deterministic processing, model flexibility, cost, compliance, and tool complexity.
| Factor | Native realtime speech-to-speech | Chained STT → LLM → TTS |
|---|---|---|
| Latency potential | Lower because the model can process and generate audio in one realtime session | More serial stages unless aggressively streamed and overlapped |
| Architecture | Fewer service boundaries | Separate transcription, reasoning, and synthesis components |
| Component control | Less ability to independently swap each speech stage | High control over STT, LLM, TTS, and domain-specific processing |
| Transcript pipeline | May require additional transcript handling depending on product needs | Transcript is a first-class intermediate artifact |
| Deterministic workflows | Works well when tool policy and output controls are designed around realtime events | Can be easier when each stage feeds strict application logic |
| Best fit | Natural, low-latency conversation and interruption-sensitive experiences | Workflows that need specialized STT/TTS, transcript processing, or independent vendor control |
Keep the speech loop short
Do not route every conversational turn through unrelated services, analytics jobs, or databases before audio can begin.
Move heavy work off the critical path
Logging, enrichment, summaries, analytics, and nonblocking writes can often happen asynchronously after the user hears the response.
Separate "need before speaking" from "can finish while speaking"
A confirmation phrase can sometimes start while a noncritical follow-up operation completes, but never speculate about a tool result that has not returned.
Design for slow dependencies
When a required API is slow, acknowledge the task, preserve state, and provide a safe wait or escalation path rather than leaving dead air.
Milliseconds need owners
How do you build a voice AI latency budget?
A latency budget assigns a target to each stage of the critical path. The numbers below are an illustrative engineering budget, not a guarantee, benchmark, or claim about a specific vendor. The point is to make every wait visible and force tradeoffs before optimization becomes guesswork.
Important
Do not add these numbers and call the sum a universal target.
Stages overlap in streaming systems, network conditions vary, and tool-backed turns may take much longer. Use this pattern to build a budget from your own traces. Report p50 and p95 separately for no-tool, read-tool, write-tool, browser, mobile, and telephony paths where those differ materially.
The cheapest latency win is often deciding sooner
How do voice activity detection and endpointing affect latency?
Voice activity detection identifies when speech starts and stops. Endpointing decides whether a pause actually means the user finished a turn. These are related but not identical problems.
OpenAI's Realtime API supports voice activity detection for detecting speech start and stop events. Current documentation includes configurable server-side VAD approaches, including semantic turn detection that uses conversational context to estimate whether the user is finished.
Shorter is faster, until it becomes rude
Aggressive thresholds reduce dead air but can interrupt people who pause naturally between phrases.
Meaning can beat a fixed timer
A contextual turn detector can distinguish "I'm done" from a pause inside an unfinished sentence more intelligently than silence alone.
Bad audio delays decisions
Background noise, echo, clipping, packet loss, and poor microphones can make speech detection less stable.
One threshold does not fit every workflow
Older callers, multilingual users, clinical intake, and complex account numbers may involve longer pauses than simple FAQ conversations.
Speech start matters as much as speech stop
The system must detect the user starting again so it can stop playback and preserve the new turn.
Measure false endpoints and delayed endpoints
Latency dashboards should include user cutoffs, immediate corrections, repeated phrases, and endpoint wait time.
Do not tune endpointing only against milliseconds. A setting that saves 150 ms but frequently cuts off names, dates, medication doses, or appointment details is a regression.
Serial pipelines create serial waiting
How does streaming reduce voice AI response time?
Streaming reduces latency by allowing the next stage to begin before the previous stage is completely finished. Audio can flow continuously. Transcription can produce deltas. A model can start reasoning on partial context. Speech output can begin before the whole response is generated.
Smaller chunks lower wait but raise overhead
Very large audio or text chunks delay downstream work. Extremely small chunks create event, serialization, and networking overhead. Measure the real tradeoff.
Keep the active prompt lean
Conversation history, retrieved knowledge, and system instructions should contain what the turn needs, not every fact the product has ever seen.
Use truthful conversational bridges
For longer tools, the agent can say it is checking availability before the result returns, as long as the preamble does not imply success.
Overlap independent reads
If location hours and appointment availability are independent, they may be fetched concurrently rather than one after another.
Distance and transport are part of the product
How do WebRTC, WebSockets, regions, and codecs affect latency?
Network latency is not something the model can optimize away. The audio path may cross a browser, mobile network, telephony carrier, media gateway, application server, model endpoint, tool provider, and back again.
| Path | Strength | Latency concern | Typical use |
|---|---|---|---|
| WebRTC | Designed for realtime media and browser/native communication | ICE negotiation, relay paths, jitter, device conditions | Web and mobile voice applications |
| WebSocket audio | Simple bidirectional application transport | Application-level buffering and TCP head-of-line behavior | Server-side realtime integrations |
| SIP | Direct fit for telephony infrastructure | Carrier routing, codecs, transcoding, PSTN path | Phone-based agents and contact centers |
| Telephony media stream | Connects live call audio to custom processing | Carrier plus WebSocket plus service hops | Programmable voice and legacy phone numbers |
Put services near the audio path
Keep your realtime gateway, model connection, and critical tools geographically sensible. Avoid unnecessary cross-region round trips.
Do not handshake on every turn
Persistent realtime sessions remove repeated setup and authentication work from the conversational path.
Transcoding can add delay
Repeated conversion between telephony and model audio formats adds compute, buffering, and quality loss.
Smooth playback has a latency cost
Jitter buffers protect against variable packet arrival, but larger buffers increase playout delay.
Twilio Media Streams can send and receive live call audio over WebSockets for near-real-time processing. WebRTC is specifically designed for realtime audio and video transport. The practical choice depends on whether users are calling by phone or speaking through your own web or mobile application.
The slowest API often becomes the voice agent
How should tool calls be engineered for low-latency voice agents?
A voice agent that only chats can feel fast. The real challenge appears when it must look up a patient, check availability, verify insurance, search a knowledge base, update a record, or send a message.
Classify the dependency
Does the result need to return before the agent can say anything useful?
Validate before calling
Do not waste a network trip on missing or invalid identifiers.
Parallelize safe reads
Run independent lookups concurrently instead of serially.
Bound the wait
Use timeouts, fallbacks, cached safe data, and escalation for slow dependencies.
Cache only what can safely be cached
Reference data and static configuration may be good candidates. Patient state or appointment availability may require a fresh check.
Keep idempotency off the critical guessing path
Retries must not duplicate bookings, messages, tasks, or records.
Use conversation state to anticipate likely needs
If the user has already selected a location, the system can prepare relevant hours or service metadata before the next turn.
Acknowledge, do not fabricate
"Let me check that" can fill a legitimate wait. "You're booked" cannot be spoken until the booking succeeds.
Silence needs an operational limit
Define how long the agent waits before retrying, simplifying the request, or escalating.
Track each dependency separately
One slow third-party API should not hide inside a single "agent latency" chart.
Fast speech is not conversational if it cannot stop
Why is barge-in latency as important as response latency?
Barge-in means the user can interrupt the agent naturally. The system detects new speech, stops or truncates the current audio response, preserves the correct conversation state, and processes the new turn.
"Actually, I need the afternoon, not the morning."
Measure time from user speech start to interruption event.
Stop queued or buffered audio quickly enough that the agent does not talk over the user.
Do not preserve words the user never heard as if they were accepted conversational context.
Do not cancel an already committed external write merely because playback was interrupted unless the business workflow allows it.
Ensure the agent's own playback does not falsely trigger user speech detection.
Include interruptions at the start, middle, and end of responses in every voice regression suite.
Phone calls add infrastructure you do not control
Why are telephony voice agents usually harder to optimize than browser voice?
Browser and native applications can often connect directly to a realtime media service using WebRTC. A phone call may add PSTN routing, carrier infrastructure, codec conversion, telephony media streaming, and your application gateway before the model receives audio.
Measure by caller region and carrier where enough volume exists.
Recognition and endpoint behavior should be tested on real call audio, not laptop microphones only.
Minimize unnecessary encode-decode cycles across the path.
Trace receive, forward, model, return, and playback timestamps separately.
Test handoff, hold, disconnect, voicemail, and caller interruption behavior.
Office Wi-Fi and synthetic audio do not reproduce mobile calls, packet loss, noisy rooms, or Bluetooth headsets.
Optimization starts with one trace per turn
What should a voice AI latency dashboard measure?
Every conversational turn should produce traceable timing events. The events should be safe for the data involved, but detailed enough to explain why one turn took 240 ms and another took 1.8 seconds.
Speech timing
speech started, meaningful speech ended, VAD decision, interruption detected, playback stopped.
Inference timing
request/event sent, first model event, first output token, first audio delta, response complete.
Dependency timing
tool selected, call started, first byte, completed, timed out, retried, rejected.
User-perceived output
first audio queued, first audio rendered, buffer depth, cancellation, underrun, completion.
Workload dimensions
transport, region, caller type, tool category, context size, audio duration, model, codec.
Quality with speed
task success, correction, repeated user phrase, escalation, abandonment, and user interruption.
Illustrative trace only. It demonstrates instrumentation and should not be presented as a measured Trilops production sample unless replaced with a verified trace from your telemetry.
Optimize in descending order of measured pain
What is the fastest way to reduce voice AI latency?
Instrument the turn
Add one shared trace ID and timestamps across capture, endpointing, model, tools, and playback before changing architecture.
Separate no-tool and tool turns
Do not let slow external APIs distort the baseline conversational path.
Tune endpointing with real audio
Reduce unnecessary silence while tracking cutoffs, repeat phrases, and correction rate.
Move to a realtime transport
Use persistent low-latency media connections instead of record-upload-wait-response cycles for live conversation.
Stream every serial stage
Process audio, transcription, reasoning, and speech output incrementally where the architecture supports it.
Shorten active context
Remove irrelevant history, precompute stable instructions, and retrieve only what the current turn needs.
Parallelize independent tools
Run safe reads concurrently and move noncritical writes, analytics, and enrichment outside the first-audio path.
Place infrastructure intelligently
Reduce cross-region hops and unnecessary media gateways between caller, application, model, and tools.
Optimize p95, then protect quality
Fix the slowest common path without increasing false endpoints, hallucinations, unauthorized actions, or user corrections.
Voice release scorecard
Latency cannot pass while conversation quality fails.
What 16+ production agents taught us
What changed when we optimized voice agents in production?
The model was not always the slowest component
Once model latency dropped, endpointing, network distance, tool APIs, audio buffering, and application logic became visible. Optimizing only inference would have missed the actual user delay.
Latency had to be measured per path, not per product
A no-tool FAQ turn, appointment lookup, write operation, transfer, and multilingual turn have different critical paths. Combining them into one average hid the engineering work.
Sub-300ms is meaningful only with a defined measurement boundary
We use the below-300ms proof point only for an optimized measured voice path, not as a blanket promise for every full conversation or external integration.
Barge-in quality changed the perception of speed
An agent that starts quickly but keeps talking after the caller interrupts still feels slow. Fast cancellation and correct conversational state are part of latency engineering.
External APIs needed conversational failure states
When a schedule or record service slowed down, leaving silence was the worst interface. A truthful preamble, timeout, retry, or escalation preserved trust without inventing a result.
The fastest architecture was not always the safest architecture
Some healthcare steps still need validation or human approval. We optimize around those controls rather than deleting them to win a latency benchmark.
Voice latency is a systems problem. The user hears the sum of every decision your architecture delayed.
Trilops production engineering principleFor the broader system architecture, read How to Build a Production-Ready AI Agent. For the healthcare voice workflow, see AI Voice Agents for Patient Intake. For release testing, see Evaluating LLMs for Healthcare.
Frequently asked questions
Voice AI latency: FAQ
What is good latency for a voice AI agent?+
There is no universal number because the measurement boundary and workflow matter. For conversational quality, measure from meaningful end of user speech to first audible agent response, then track p50 and p95. Also measure cutoffs, barge-in, tool completion, and task success.
Can a voice agent respond in under 300 milliseconds?+
Specific optimized paths can reach below 300 milliseconds under defined conditions. That should not be interpreted as a guaranteed full-turn result across every network, telephony route, tool call, model, or geography. Publish the measurement boundary with the number.
Is speech-to-speech always faster than STT plus LLM plus TTS?+
Native realtime speech-to-speech removes some serial boundaries and can offer excellent latency. A carefully streamed chained architecture can still be appropriate when you need specialized transcription, synthesis, deterministic transcript processing, or independent vendor control.
What usually causes the biggest delay in a voice agent?+
It depends on the path. Common causes include slow endpoint detection, network distance, model response time, external tools, large context, audio buffering, telephony routing, and retries. Instrument every stage before optimizing.
How do tool calls affect voice AI latency?+
A required tool can dominate the turn if it takes hundreds of milliseconds or seconds. Parallelize independent reads, prefetch safe data, use bounded timeouts, and speak truthful preambles where appropriate. Never announce a tool result before it succeeds.
Why does my voice agent feel slow even when model latency is low?+
The model may be only one part of the delay. Endpointing can wait too long, the client may buffer audio, the network can add round trips, or the agent may fail to stop quickly during interruption. User-perceived latency must be measured end to end.
Should we optimize p50 or p95 first?+
Establish a healthy p50 baseline, then focus heavily on p95 because slow outliers damage trust and often reveal architectural bottlenecks. Keep quality metrics alongside latency so optimization does not increase cutoffs, wrong actions, or corrections.
Authoritative references
- OpenAI API: Realtime and audio
- OpenAI API: Realtime API with WebRTC
- OpenAI API: Voice activity detection
- OpenAI API: Voice agents
- Twilio: Media Streams
- WebRTC project: realtime communication
Latency depends on architecture, network, model, region, transport, endpointing, tools, audio format, traffic, and measurement boundaries. Illustrative budgets and traces in this article are engineering examples, not universal performance guarantees.
Do not optimize the demo. Optimize the conversation path.
Trilops builds production voice agents with streaming audio, measurable latency budgets, controlled tools, interruption handling, evaluation, observability, and healthcare-ready workflow safeguards.
Let's start a project together