鉴于如今有如此多的解决方案（包括 LiveKit），而且用 AI 编写代码已经变得如此简单，几乎任何人都能在几分钟内跑起一个演示用的语音智能体。事实上，每个月都有数万名开发者（以及一些非开发者）在 LiveKit 上搭建并运行智能体。
不过，本指南并不只是帮你把语音智能体跑起来。它的目的是分享我们的解决方案架构师和前线部署工程师多年来积累的经验——他们帮助客户将 POC 转化为真正投入生产、承载数百万分钟 AI 语音智能体对话的线上智能体。
AI 语音智能体的生产就绪度归结为 3 个问题#
在接下来的章节中，我们将带你逐一了解应对每一类需求的建议，以及我们的工程师在实现 AI 语音智能体过程中总结的一些技巧。
- 智能体是否足够聪明、听起来足够自然，能够与我们的用户对话？
- 我们能否在生产环境中弄清楚它的表现如何？
- 我们构建它的方式，是否能让我们可靠且低成本地扩展它？
构建聪明且听起来自然的语音智能体的 7 种方法#
1. 尽可能优化智能体的首次响应#
我大学时上网球课时，教练说，如果你想变得出色，就应该把大部分练习时间花在一个动作上：发球。这是唯一一个从开始到结束都由你掌控的击球，而且它为这一分的其余部分奠定了基础。
你的语音智能体的首次响应也是同样的道理。把通话最初几秒做对，是我们一位前线部署工程师从即将上生产的团队那里听到的头号关注点。而这一点也得到了我们在实际场景中所见数据的有力支持。经过测量，一家公司发现大约四分之三的通话在智能体的问候环节就结束了。
我们的建议归结为以下几点：
单独测量第一轮。把从接听到智能体说出第一个字的时间作为独立指标来跟踪，与你的 p50 和 p95 轮次延迟分开。当我们深入分析一个团队缓慢的开场时，问题并不在模型。在智能体开口之前，它一直在等待一个 HTTP 调用来选择开场白，仅这一项就花了将近整整一秒。把它移开，是我们发现过的最廉价的延迟优化之一。
只加载智能体说“你好”所需的内容，其余内容在它说话时再获取。把你的上下文分成两个桶。第一个桶存放良好问候所需的最低限度内容，通常只是来电者的姓名。通过 作业元数据、房间元数据或 SIP 头随通话一起传入，这样智能体就永远不必等待查询。与此同时，第二个桶存放其他所有内容，并在智能体问候来电者时开始加载。
1# This code shows how to greet the caller by name while their full record loads in the background2
3import asyncio4import json5
6from livekit.agents import Agent, AgentServer, AgentSession, JobContext, function_tool7
8
9class SupportAgent(Agent):10 def __init__(self, customer_record: asyncio.Task[dict]) -> None:11 super().__init__(instructions="You are a friendly support agent. Keep replies short.")12 self._customer_record = customer_record13
14 @function_tool()15 async def get_account_details(self) -> dict:16 """Look up the caller's account, plan, and recent orders."""17 return await self._customer_record # usually loaded long before the caller asks18
19
20server = AgentServer()21
22
23@server.rtc_session(agent_name="support-agent")24async def entrypoint(ctx: JobContext):25 # Bucket one: just enough to greet, passed in with the dispatch26 metadata = json.loads(ctx.job.metadata)27
28 # Bucket two: start the slow lookup now, but don't wait for it29 customer_record = asyncio.create_task(load_customer_record(metadata["customer_id"]))30
31 session = AgentSession() # stt, llm, tts32 await session.start(room=ctx.room, agent=SupportAgent(customer_record))33 session.generate_reply(34 instructions=f"Greet {metadata['first_name']} by name in one short sentence."35 )
提示让第一句话保持简短，并把重要的部分放在前面。
2. 像人一样处理轮次转换和打断#
如果 AI 语音智能体（或者人，就这点而言）总是在错误的时机打断你，或者继续喋喋不休地讲与你无关的内容，那会极其令人沮丧。为了应对这种糟糕的体验，大多数将智能体投入生产的团队都会使用轮次转换模型，例如 LiveKit 的轮次检测器, 并配合 自适应打断处理 这样一句简短的“嗯哼”就不会打断智能体。
只需一行： turn_detection=inference.TurnDetector()。它也是默认值，因此即使你省略这一行， AgentSession 也会将其开启，并且只要你的语音转文字模型支持，它就会加入自适应打断处理。
遗憾的是，一个设置后即可不管的轮次转换模型并不总能完美适配所有用例。我们的解决方案架构师和前线部署工程师建议，在发布前用你支持的每种语言测试句子中间的长停顿、像“是”或“否”这样的简短回答以及打断情况。如果你发现任何缺口，只覆盖能修复你所听到问题的那个设置，其余一切保持默认：
1# This code shows how to turn on LiveKit's turn detector and tune it only if testing finds gaps2
3from livekit.agents import AgentSession, TurnHandlingOptions, inference4
5session = AgentSession(6 # ... stt, llm, tts7 turn_handling=TurnHandlingOptions(8 # LiveKit's turn detector (on by default)9 turn_detection=inference.TurnDetector(),10 # Optional: uncomment only to fix a gap your testing finds11 # endpointing={"min_delay": 0.6}, # wait longer when callers pause mid-thought12 # interruption={"min_words": 2}, # don't stop for a cough or a lone "yeah"13 ),14)
有关完整选项列表以及每个选项能修复哪种症状，请参阅 轮次转换调优指南 或 在 LiveKit Agents 中配置轮次检测和打断.
提示如果你手边没有母语者来测试停顿， LiveKit Agent Simulations 的语音功能是一个不错的起点。
3. 避免不自然的沉默#
人类客服在电话中花一秒钟思考答案、查资料或采取行动，这很正常。但当他们停顿时，你通常确信他们仍在线上，而且他们通常清楚自己在做什么。
AI 智能体也需要一些时间来思考、查资料或采取行动（尤其是当它们与遗留系统协作时），但你如何调优智能体来处理并尽量减少沉默，可能会带来很大差别。
实现这一点的三种最常见方法是：
使用能高效完成任务的快速 AI 模型。加入你熟悉的 LLM（比如 OpenAI 或 Anthropic 的最新模型）并不总是正确的选择。事实上，在许多情况下，我们的解决方案架构师并不建议使用它们。例如， LiveKit Inference 上的 Gemma 4 31B 往往能以远少于通用模型的思考时间和延迟得出正确答案并完成工具调用。我们的几位客户从 GPT-4.1 切换到 Gemma 后，在 p50 上节省了 300–500 毫秒，同时智能体仍能通过所有评估。你可以在 LiveKit Inference 延迟基准测试.
上自行比较各模型。相同请求的首 token 时间，来自 Gemma 4 发布基准测试 （2026 年 7 月）。越低越快。
- Gemma 4 31B · LiveKit192 毫秒
- GPT-5.5966 毫秒
- GPT-4.11,006 毫秒
- Gemini 3.0 Flash1,095 毫秒
以异步方式调用长时间运行的工具。我们的前线部署工程师与各种规模的公司合作，其中许多公司都遇到过响应需要数秒的系统。当你知道某次查询会耗时较久时，可以将该工具设为 异步 ，这样智能体就能在工作运行的同时继续对话。做得好时，它可以比人类更好地多任务处理。以下是使用异步查询时一次通话的展开过程：
1sequenceDiagram2 participant C as Caller3 participant A as Agent4 participant S as Claims system5 C->>A: "What's the status of claim 4417?"6 A->>S: Starts the lookup in the background7 A->>C: "Looking up claim 4417 now."8 C->>A: "While you're at it, can I update my address?"9 A->>C: Helps with the address10 Note over C,A: 5 seconds of silence11 A->>C: "Still pulling that up, one moment."12 S-->>A: Claim status13 A->>C: "Claim 4417 was approved yesterday."
1# This code shows how to keep talking during a slow lookup instead of leaving dead air2
3from livekit.agents import RunContext, function_tool4
5
6@function_tool()7async def check_claim_status(ctx: RunContext, claim_id: str) -> str:8 """Look up a claim in the claims system. This can take 10 seconds or more."""9 # The first update makes the tool non-blocking: the agent acknowledges the request,10 # and the conversation keeps going while the lookup runs in the background11 await ctx.update(f"Looking up claim {claim_id} now.")12
13 # If the line goes quiet for 5 seconds, the agent says this instead of leaving dead air14 async with ctx.with_filler("Still pulling that up, one moment.", delay=5):15 claim = await claims_api.get_claim(claim_id)16
17 # Voiced as a follow-up reply once the lookup finishes18 return f"Claim {claim_id} is {claim.status}."
注意提示词大小和缓存。我们的解决方案架构师和前线部署工程师协助解决响应缓慢问题时，最省成本的修复方法之一就是优化提示词大小并正确缓存。作为参考，小提示词（约 2k token）与大提示词（超过 15k token）之间的差异，在 p50 上可达约 500 毫秒，在 p95 上可达整整一秒。缓存可以简单到只是重新排列提示词：我们的一项内部测试发现，仅将静态内容移到系统提示词顶部，延迟就改善了 38%。
提示如需了解更多在整个流水线中查找并削减延迟的方法，请参阅 理解并改善语音智能体 延迟.
4. 帮助你的智能体倾听客户，而不被背景噪音分散注意力#
It's pretty clear to a human whether the person on the other end of the call is talking or it's background noise from Love Is Blind: Sweden on their TV. To an agent, it's not always clear, and we've seen cases where an agent has a very confused conversation with someone's background TV or accidentally cancels out the user's voice altogether. There are a number of ways to guard against this in production, and we've documented them in LiveKit noise cancellation: what it is and how it works.
TipOne of the things we see with teams going into production is that they didn't test the agent in
background noise scenarios. You can run your scenarios with background
noise on
LiveKit using lk agent simulate audio --scenarios scenarios.yaml --background-noise.
5. Give your voice agent the right accent for your region and expressions for the context#
There's something uncanny about a voice agent that delivers every line in a deadpan monotone and never takes a breath, or a voice agent for a Mexican bank speaking with an accent from Spain. These issues tend to show up late when you're delivering a multilingual agent meant to span many regions.
Getting a voice agent to be expressive on LiveKit is easy if your TTS provider supports it. Turn on expressive mode and the LLM adds emotion, pacing, and non-verbal sounds inline, matched to the moment:
1# This code shows how to make the agent's voice expressive with a single setting2 3from livekit.agents import AgentSession, inference4 5session = AgentSession(6 stt=inference.STT(model="assemblyai/universal-3-6-pro", language="en"),7 llm=inference.LLM(model="google/gemma-4-31b-it"),8 # Needs a LiveKit Inference TTS that supports expressive mode9 tts=inference.TTS(model="inworld/inworld-tts-2", voice="Ashley"),10 # The LLM adds emotion, pacing, and breaths. Tags are stripped from the transcript.11 expressive=True,12)
Try it yourself: tell the agent some good news or some bad news, then toggle expressive mode off to hear the same agent flat.
TTS ProviderExpressionTell it some good news, or some bad news, and listen to how the delivery changes.
Getting the right accent can be a bit trickier. The first place our team generally recommends you focus your testing is picking a voice built for the locale. It sounds obvious, but it's important to know whether your vendor supports all the locales you need so you can switch automatically when needed for a bilingual agent.
1# This code shows how to build a bilingual agent that replies in the caller's language2 3from livekit.agents import Agent, AgentSession, inference4 5session = AgentSession(6 # "multi" transcribes English and Spanish, including mid-conversation switches7 stt=inference.STT(model="deepgram/nova-3", language="multi"),8 llm=inference.LLM(model="google/gemma-4-31b-it"),9 # A Mexican Spanish voice. Leaving `language` unset lets the TTS detect it on each reply.10 tts=inference.TTS(model="inworld/inworld-tts-2", voice="Diego"),11)12 13agent = Agent(14 instructions="Reply in the caller's language. When speaking Spanish, use Mexican Spanish."15)
If you need tighter control, such as a different voice per language, see Build a multilingual voice agent that automatically switches languages.
TipIn some cases, matching the entire STT, LLM, and TTS pipeline to the right language and locale helps. For example, a phone number spoken in British English can be transcribed differently from one spoken in American English, which affects how the agent reads the number back.
6. Intelligently clean up spelled-out information#
The key here is "intelligently." Nothing is worse than panic-spelling "M as in Michael, I as in icicle, C as in cat?" It shouldn't take five minutes of back-and-forth with your agent to get the right information pulled up.
Our team's approach usually has three parts:
Hear the spelling correctly without cutting the caller off. Spelling is full of pauses, and an agent that treats every gap as the end of a turn will jump in after "M-I-C." LiveKit's turn detector understands that a caller mid-spelling isn't done talking, and you can give it a little more patience (a higher min_delay) for the moments that need it.
Clean it up with a dedicated step instead of the main prompt. Don't make your main prompt do this job. LiveKit's prebuilt tasks (GetNameTask, GetEmailTask, GetPhoneNumberTask, GetAddressTask, and more) are built for exactly this: they turn "john dot doe at g mail" into john.doe@gmail.com and remember what the caller already said so they never ask twice. Additionally, you can add an agent tool that fuzzy-matches against your records for some cases.
1# This code shows how to collect a spelled-out email address in a dedicated step and confirm it2
3from livekit.agents import Agent, function_tool4from livekit.agents.beta.workflows import GetEmailTask5
6
7class SupportAgent(Agent):8 @function_tool()9 async def collect_email(self) -> str:10 """Collect the caller's email address to look up their account."""11 # Normalizes "john dot doe at g mail" and reads it back once to confirm12 result = await GetEmailTask(chat_ctx=self.chat_ctx)13 return f"The caller's email is {result.email_address}."
Read it back once in a form the caller can check. In many cases, one clean confirmation beats three rounds of guessing, so our team often recommends having the agent read it back letter by letter, saying "dot" and "at" out loud: "That's j o h n dot d o e at gmail dot com, right?"
And for anything numeric, like account numbers, PINs, or confirmation codes, let callers use the keypad. LiveKit agents accept keypad presses and spoken digits interchangeably, so nobody has to read a 12-digit number out loud.
7. Keep your agent on script#
In March 2026, someone posted that Chipotle's chatbot would happily write Python. They asked it to reverse a linked list before placing an order, and the bot explained the algorithm, wrote working code, then politely asked what they wanted for lunch.
It's pretty funny until it's your brand (and your AI inference budget). Voice agents face the same thing: within the first day of launch, callers may start asking off-topic questions, pushing for exceptions, or trying the classic "ignore your previous instructions." And voice can be a bit trickier than text. In edge cases, your agent can slip in ways callers hear: reading a function call aloud, switching languages mid-sentence, or rambling through a 30-word answer to a yes-or-no question.
Here are four ways we've helped teams tackle this:
Write the prompt for the model you're running. Every model goes off the rails in its own way, so a prompt that works on one won't always carry over to another. In our experience, fast models like Gemma do best with explicit, prescriptive rules ("do this, never do that"), while larger general-purpose models do better when you describe the goal and let them work out the steps. Whichever you use, name the specific things it must never say out loud. For subtler slips, such as switching languages, use LiveKit Observability to trace the turns where the slip happens, track how often it happens, and tighten the prompt around those turns.
TipBe careful with examples in your prompt. Models tend to over-rotate on them, repeating the example phrasing word for word or forcing every conversation into the example's shape.
Validate in code instead of the system prompt (when it makes sense). The more rules you put in the prompt, the more chances the model has to get one wrong. Anything that has a right answer belongs in a tool. Verifying a date of birth is a good example: LLMs are notoriously bad at dates, so instead of asking the model to compare "March 3rd, '85" to a record, have it call a verify_date_of_birth tool and let your code return a yes or no. The same goes for eligibility checks, account lookups, and business rules.
1# This code shows how to check a date of birth in your code instead of trusting the model2 3from datetime import date4 5from livekit.agents import Agent, function_tool6 7 8class VerificationAgent(Agent):9 def __init__(self, dob_on_file: date) -> None:10 super().__init__(11 instructions=(12 "Ask for the caller's date of birth, then call verify_date_of_birth. "13 "Don't discuss the account until it returns Verified."14 ),15 )16 self._dob_on_file = dob_on_file17 18 @function_tool()19 async def verify_date_of_birth(self, year: int, month: int, day: int) -> str:20 """Check the caller's date of birth against the account on file.21 22 Args:23 year: Four-digit birth year, for example 1985.24 """25 # The model only extracts the date. Your code decides whether it matches.26 dob = self._dob_on_file27 if (year, month, day) == (dob.year, dob.month, dob.day):28 return "Verified."29 return "Not verified."
Say less. LLMs are trained to be thorough, which is fine in a chat window but feels endless on a call. Not to mention every extra word costs money. On one forward deployed engagement, our engineers compared the agent with the competitor it was replacing and found it averaged about five more words per turn. Trimming wordy recovery steps turned out to be the biggest lever on total call length.
For high-stakes calls, let a second model keep watch. In some rare cases where you need a conversation to be very fast but you can't sacrifice the intelligence of a smart model, our forward deployed engineers recommend an observer pattern: a fast model handles the conversation, while a smarter model listens in the background and quietly steers the agent's next turn if needed. The observer doesn't block speech, so callers never hear the checking happen. There are major tradeoffs to consider, though. For example, the observer can only correct the next turn, not the current one; running two models costs more; and the added complexity is usually a stopgap, not a long-term design.
3 ways to set yourself up for success observing your voice agent in production#
Shipping your agent to production, even to a small slice of traffic, can be both an exciting and nerve-wracking experience. No matter how many scenarios you've tested or best practices you've followed, once you launch, you're bound to find opportunities for improvement from a latency, intelligence, or conversational flow perspective.
The teams that have the smoothest first week in production decide ahead of time what they're allowed to record and what "good" looks like, and they have everything set up so they can quickly root-cause a problem when a call doesn't go perfectly.
1. Decide what you're allowed to record before you record it#
Make sure what you're recording follows your data residency, PII, and data retention requirements.
Double-check your observability region (if needed). If you're using LiveKit Cloud for observability, your project's data region decides where traces, logs, metrics, transcripts, and recordings are stored. Today that's the United States (the default) or the European Union (Frankfurt). It's set when you create the project.
Turn on PII redaction (if necessary). LiveKit can remove personal information from transcripts, audio, and telemetry before it's stored: spoken details become a beep, and text becomes a redaction marker. It's off by default and included with LiveKit Observability at no extra cost.
2. Hypothesize what "good" looks like before you launch (and make sure you're set up to measure it)#
Voice agents can be complex systems, and it's easy to get "lost in the logs" after deployment. To help focus your first days in production, we recommend focusing on a few things.
How fast it feels. Our engineers typically look at end-to-end latency at p50 and p95, plus the first turn of the call. LiveKit Observability reports e2e_latency on every agent turn. You can inspect it call by call in Agent Insights (it's the latency under each agent reply in the screenshot above), see p50/p95/p99 in audio simulations before you launch to get a rough idea, and export it to the tools you already use to track percentiles across all your production calls.
Whether it does the job. Pick an outcome that matches what you built. The goals for an outbound agent, a customer service agent, and an AI interviewing agent differ, so there isn't a one-size-fits-all answer. Whatever you choose, our FDEs suggest making it something you can count, rather than something you can only judge from transcripts. If success means handing a qualified lead to a person, the transfer_to_human tool call is your conversion event. Track what happens after the handoff too, such as whether the lead arrives warm or cold.
If you do use an LLM to grade calls, keep the scale small. We've found models are inconsistent graders on a 0–100 scale, but they tend to do better on a 0–7 or binary scale.
3. Make sure you can find and cluster edge cases from your transcripts#
It's common when developing voice agents to fall into testing the same conversations over and over until they're all consistently successful, but in nearly every case we've seen, you'll inevitably find edge cases when you go to production.
Our forward deployed engineers usually start by having an LLM sort the first few days of transcripts into clusters, then work through the clusters of edge cases one at a time, by priority.
3 ways to build a reliable and cost-efficient voice agent in production#
1. Assume every model will fail at some point#
Every model provider has outages; it's not an "if," it's usually a "when." When deploying at production scale, we recommend including fallbacks for each model. If you're using an STT-LLM-TTS pipeline, that means a fallback for each stage. In the LiveKit Agents framework with LiveKit Inference, this is usually a quick line or two. And don't stop at the STT-LLM-TTS pipeline: consider your tools as well if they make LLM calls.
1# This code shows how to add a backup for every model so a provider outage doesn't end the call2
3from livekit.agents import AgentSession, inference, llm4
5session = AgentSession(6 # STT and TTS fallbacks run server-side in LiveKit Inference7 stt=inference.STT(8 model="assemblyai/universal-3-6-pro",9 fallback=[{"model": "deepgram/nova-3"}],10 ),11 # LLM fallbacks run in your agent process, tried in order12 llm=llm.FallbackAdapter(13 [14 inference.LLM(model="google/gemma-4-31b-it"),15 inference.LLM(model="openai/gpt-4.1-mini"),16 ]17 ),18 tts=inference.TTS(19 model="inworld/inworld-tts-2",20 voice="Ashley",21 fallback=[{"model": "cartesia/sonic-3", "voice": "9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"}],22 ),23)
TipBefore launch, "break" your primary provider on purpose in staging: give it a bad API key or block its hostname, then run a few test calls, including one where you break it mid-conversation. If what you hear is silence instead of a slightly different voice, your fallback isn't done.
2. Make sure your backend is set up to handle scale#
It's logical to think you need to run a massive load test before launching, but our forward deployed engineers, who have helped enterprises deploy hundreds of voice agents at scale, usually say it isn't necessary.
LiveKit Cloud scales with you, so a load test of the media and agent layers can end up being a waste of budget. Instead, they recommend testing each of the pieces you own and making sure they're set up correctly and with the right limits: your backend databases can handle many connections, your APIs can handle the request volume, and, if you're not using LiveKit Inference, your model providers are set up for your concurrency expectations.
3. Know what a conversation costs before you scale it#
It's tempting to build your voice agent with all the freshest models you're familiar with in your daily life, but when you scale that up to tens or hundreds of thousands of minutes a month, costs add up. And when you go back to switch out models, you might need to add new tools, guardrails, or prompts, because every model has its own quirks.
TipAs of October 2026, our team generally recommends Gemma 4 31B on LiveKit Inference. It's fast, good at tool calling and following instructions, and very cost-effective, coming in about 5–7x cheaper per token than GPT-4.1. See the voice model capability benchmarks and Inference pricing to compare.
Don't let this list keep you from shipping#
We packed a ton of advice into this guide. Every recommendation here comes from real deployments: conversations with enterprise teams, weeks spent digging through call data, and fixes that made a measurable difference in production. But it's also specific. Some of it will matter a lot for your use case, and some of it won't matter at all.
So don't treat this as a checklist you have to finish before launch. Pick the few items that matter most for your agent, get it in front of real traffic, and let actual calls tell you what to fix next. The best voice agents we've seen weren't perfect on day one. They shipped, learned quickly, and kept getting better.
If you hit something you can't solve along the way, we'd love to help. Our developer success engineers, solution architects, and forward deployed engineers help teams take voice agents from POC to production every day. Talk to our team or explore the LiveKit Agents docs to get started.