在 Amazon SageMaker AI 上用 Kokoro TTS 为 AI 智能体流式输出低延迟语音

人工智能

在 Amazon SageMaker AI 上使用 Kokoro TTS 从 AI 智能体流式输出低延迟语音

客户向 AI 语音智能体提出一个问题，然后等待。智能体其实已经有了答案，它正在生成一段很长的、包含多个段落的回复，但客户有五秒钟什么都没听到。等到答案传来时，客户已经感到沮丧、困惑，也不再关注了。

文本转语音集成恰恰造成了这种差距。一句回复可能只需 600 毫秒就能生成，但完整回复需要五秒钟，而集成会等全部内容生成完毕后才开始合成。

本文介绍如何弥合这一差距。智能体每生成一句话，我们就立即将该句流式传输到托管在 Amazon SageMaker AI 上的 Kokoro-82M 文本转语音（TTS）模型，因此合成从第一句就开始，而不是等到完整回复生成完毕。把音频传回浏览器也需要专门的解决方案，因此我们还会介绍实现这一目标的两种传输方式，以及它们在实际应用中的各自代价。

解决方案概述

该解决方案通过在生成的同时进行流式合成，而不是等待生成完成，来弥合这一差距。智能体每生成一句话，该句就会立即流式传输到托管的 Kokoro-82M 文本转语音模型，因此音频播放可以在智能体说完完整回复之前就开始。Kokoro-82M 仅有 8200 万个参数，小到足以在单台中端 GPU 实例上低成本运行。这使得处理真实流量的语音智能体能够同时保持低延迟和低推理成本。

Kokoro-82M 在 SageMaker AI 实时端点上的自定义容器中运行，部署在 ml.g5.xlarge GPU 实例。该容器暴露两条路由： POST /invocations，用于为单个完整输入流式返回响应；以及 WS /invocations-bidirectional-stream，用于在双向通信中通过一条保持打开的连接传输多条消息。

下图展示了从客户端到模型的请求路径：

图 1：客户端通过双向流与 SageMaker AI 实时端点之间的请求路径

先决条件

开始之前，您需要：

• 一个具有创建 SageMaker AI 端点权限的 AWS 账户；AgentCore 运行时（Amazon Bedrock AgentCore 的一项能力）；AWS Identity and Access Management（IAM）角色；以及一个 Amazon Elastic Container Registry（Amazon ECR）存储库，至少包含 sagemaker:CreateEndpoint, sagemaker:CreateEndpointConfig, sagemaker:CreateModel, sagemaker:InvokeEndpointWithBidirectionalStream和 bedrock:InvokeModelWithResponseStream 操作。
• 示例容器固定使用了一个较旧版本的 PyTorch，该版本存在一个已知安全公告（PYSEC-2026-2286，已在 torch 2.10.0 中修复）。在任何非概念验证用途之前，请审查并升级该固定版本。
• AWS Cloud Development Kit（AWS CDK）v2，已安装并在您的目标账户和 AWS 区域中完成引导。
• 一个从声明了 com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true Docker 标签的容器部署的 SageMaker AI 实时端点。
• Python 3.12 或更高版本、 uv，以及 Docker，用于构建容器镜像（底层 HTTP/2 客户端仅为 Python 3.12 及更新版本提供预构建 wheel）。
• 适用于 Python 的 AWS SDK（Boto3），并支持 invoke_endpoint_with_bidirectional_stream.
• 熟悉 WebSockets 以及 Starlette 或类似的 ASGI 框架。
• 来自 GitHub 存储库的部署代码 中引用的内容。

演练

本演练先部署端点和智能体，然后逐步说明客户端侧的改动，这些改动使增量流式传输能够端到端工作。

部署端点和智能体

Kokoro-82M 端点和智能体运行时通过两个 CLI 工具部署，每个工具都封装了各自的 AWS CDK 堆栈：

uv run speech-infra deploy kokoro-82m
uv run agent-infra deploy --bedrock-model-id <your-bedrock-model-id>

speech-infra deploy 负责部署 Kokoro-82M 端点。这包括基础堆栈和模型缓存堆栈，它们创建底层 IAM 角色并将模型权重同步到 Amazon Simple Storage Service（Amazon S3）。随后，端点堆栈构建容器镜像，并创建 SageMaker AI 模型、端点配置和端点。 agent-infra deploy 负责部署托管 Strands 智能体的 Amazon Bedrock AgentCore 运行时，该智能体会调用您刚刚部署的端点，因此请先部署端点。

第 1 步：处理 SageMaker AI 实际发送的 WebSocket 帧类型

SageMaker AI 使用的双向传输会将客户端消息的每一部分作为二进制 WebSocket 帧传送到您的容器。编写 Starlette WebSocket 处理程序的一种常见方式是调用 receive_text()，它会读取 message["text"] ，并在该键不存在时抛出 KeyError 。由于此客户端发送的每条消息都以二进制形式到达，该处理程序会在每个请求上失败。

这种失败常被误诊。一个将 str(e) 转发给客户端的通用异常处理程序会把 KeyError('text') 变成一条错误帧，内容为 'text'，这看起来像是缺少字段，而不是帧格式问题。端点本身报告 InService 始终如此，因为它接受连接并响应每条消息，每次都会返回一个错误。

显式接受任一帧类型，而不是假定会出现某一种：

from starlette.websockets import WebSocket, WebSocketDisconnect

async def _receive_message(websocket: WebSocket) -> str:
    message = await websocket.receive()
    if message["type"] == "websocket.disconnect":
        raise WebSocketDisconnect(message.get("code", 1000))
    payload = message.get("text")
    if payload is None:
        payload = message.get("bytes", b"").decode("utf-8")
    return payload

当没有文本字段时，它会将字节解码为 UTF-8，而不是假定某一种帧类型。通过发送二进制帧来测试这一点，因为这种故障模式会产生格式正确的错误帧和健康的端点状态，而不是明显的崩溃。

第 2 步：在消息之间保持会话打开

当你的容器完成一条消息的音频合成后，本能反应是关闭会话，就像 HTTP 周期结束那样。但在这里这是错误的。在发送一条消息的音频和一个 synthesis_complete 控制帧之后，容器循环应回到等待 receive() 的状态，以在同一仍然打开的会话上接收下一条消息。仅当客户端发送显式的 {"type": "close"} 消息时才关闭连接。

下图展示了一个会话按顺序承载两个句子，仅在最后关闭：

图 2：一个双向会话按顺序承载两个句子，仅在最后关闭

这样，你就可以在一个持久会话上按顺序发送多个句子，而它们之间没有重连开销。在一次响应后急于关闭会迫使每个句子都建立新连接，从而抵消了大部分延迟收益。

第 3 步：在智能体生成句子时流式传输它们

With the session behavior in place, the client-side fix for the latency problem is a class that normalizes text arriving in two shapes: a complete string known up front, or a live token stream. It produces the same output either way: complete sentences, ready to send as soon as each one is complete.

import re

_SENTENCE_RE = re.compile(r"(?<=[.!?])\s+")


class IncrementalSentenceChunker:
    def __init__(self) -> None:
        self._buffer = ""

    def feed(self, fragment: str) -> list[str]:
        self._buffer += fragment
        parts = _SENTENCE_RE.split(self._buffer)
        if len(parts) == 1:
            return []
        self._buffer = parts[-1]
        return [s.strip() for s in parts[:-1] if s.strip()]

    def flush(self) -> str | None:
        remainder = self._buffer.strip()
        self._buffer = ""
        return remainder or None

Call feed() once per fragment of text as it arrives. It returns any newly completed sentences, holding back trailing text. Call flush() once, at the end of input, to release whatever partial text remains even without trailing punctuation. This keeps a stream that ends before a period arrives from dropping a final sentence.

This isn’t concurrent synthesis. Every bidirectional handler is a strictly sequential loop: receive a message, synthesize its audio, send that audio, send synthesis_complete, then receive the next message. Sentence two’s audio isn’t generated while sentence one’s still transmits. What incremental sending buys is narrower: synthesis on sentence one begins the instant it exists, rather than waiting for the full response. For a multi-sentence response with real generation time between its first and last sentence, that’s the entire latency win.

We verified this against a live endpoint sending a three-sentence input over one session. All three chunks arrived with audio matching the text sent for each:

Chunk	Text	Audio bytes	Duration (s)
1	“Hello there.”	73,200	1.525
2	“This is a post deployment check.”	108,000	2.250
3	“It has three sentences.”	98,400	2.050

Step 4: Choose a transport for getting audio back to the client

Streaming text into the model incrementally solves half the problem. The other half is getting the resulting audio out to whatever renders it, commonly a browser. Most agent and chat frontends already stream text over Server-Sent Events (SSE): one-directional, and supported natively by most browsers. SSE is a text protocol, not built to carry a raw binary payload. A team adding voice to an existing SSE pipeline naturally reaches for the pattern it already has: base64-encode the audio and wrap it in a JSON event.

We measured the cost of that approach across 24 clips, confirmed with Amazon Transcribe that quality didn’t differ across encodings, and compared MP3 against AAC:

Candidate	Raw bytes	Base64 bytes	Bitrate
Raw PCM	2,396,400	3,195,200	384 kbps
MP3 48k	308,304	411,072	49.5 kbps
AAC 32k fMP4	258,039	344,084	41.6 kbps

Base64 inflates every payload by close to the expected 33%. The subtler cost is frame alignment. SSE events are delimited by a blank line, but the audio inside is delimited into MP3 frames. The two schemes don’t necessarily align with how the transport chunks the data. In one measured response, four logical MP3 frames arrived as six delivery chunks, two ending mid-frame. A consumer that treats each delivery chunk as one complete unit fails on ordinary, healthy traffic. The fix buffers incoming bytes and acts only once a delimiter is found:

buffer = b""
for event in resp["Body"]:
    if "PayloadPart" not in event:
        continue
    buffer += event["PayloadPart"]["Bytes"]
    while b"\n\n" in buffer:
        frame, buffer = buffer.split(b"\n\n", 1)
        handle(frame)

The alternative used today for the model-facing leg skips base64 and SSE framing entirely. It multiplexes raw binary audio and JSON control frames over the same bidirectional session already carrying incremental text input, and tells them apart with one rule.

import json


def _decode_control_frame(chunk: bytes) -> dict | None:
    try:
        decoded = json.loads(chunk.decode("utf-8"))
    except (UnicodeDecodeError, json.JSONDecodeError):
        return None
    return decoded if isinstance(decoded, dict) else None

A chunk that fails to decode as a JSON object is audio, forwarded as received, with no encoding step anywhere in the path. One edge case matters: raw PCM audio can legitimately begin with byte 0x7b, the ASCII character {, the same character that opens a JSON object. This rule avoids misclassifying that audio because it requires the entire chunk to parse as valid JSON, not its first byte alone, something a PCM frame will essentially never do.

Note: If your frontend already has a working SSE pipeline for text, base64-encoded audio inside an SSE event is the pragmatic choice. It costs roughly 33 percent extra payload and a buffer-and-reassemble step, but reuses infrastructure you already operate. If you’re building the voice path fresh, or latency is the priority, the unified binary and JSON approach avoids both costs and reuses the connection already carrying text.

Step 5: Put it together in a real agent

The fuller, end-to-end case is a live, generative agent composing its response one token at a time, with no way to know sentence four until sentence one exists. This example runs as a Strands agent, backed by Amazon Bedrock, deployed to Amazon Bedrock AgentCore runtime, calling the Kokoro endpoint with the pattern from Step 4. Three communication legs connect in this design: the browser-facing leg carrying Server-Sent Events back to the caller, the agent’s streaming call to Amazon Bedrock for text generation, and the agent’s bidirectional session with the Kokoro endpoint for synthesis.

The following diagram shows those three communication legs, and where the overlap between generation and synthesis happens:

Figure 3: The three communication legs connecting the caller, Amazon Bedrock, and the Kokoro endpoint

The request payload carries a mode field, either "bidi" or "batch", dispatched to two code paths inside the same entrypoint function. A benchmark between them changes exactly one variable: when synthesis starts. In run_bidi, two tasks run concurrently. One drains the Bedrock text stream and feeds completed sentences into the chunker. The other drains the Kokoro session and emits audio chunks as they arrive. Both push into one shared queue, so a consumer yields events in true arrival order: audio for an early sentence can reach the caller while Bedrock still generates a later one. run_batch is the deliberate opposite: it waits for the full response before one synthesis call, kept as a fair baseline.

Going the other direction, the leg from the agent container back to the caller does use base64, for a different reason than Step 4 describes. BedrockAgentCoreApp’s entrypoint must yield plain, JSON-serializable dictionaries, which its SDK turns into data: {json}\n\n frames automatically. There’s no way to hand it a raw binary frame. That’s a consequence of the JSON-native managed runtime, not the choice Step 4 measured and moved away from. Both run_bidi and run_batch pay that identical cost on this leg, so the measured difference between them is purely timing: when the first audio chunk is emitted.

Testing and validation

We isolated the TTS transport’s own contribution to time to first audio by sending the same 267-character, four-sentence block of text two ways: as one message (batch), and as only its first sentence (incremental). We ran five trials each against the live endpoint.

As measured earlier, sending only the first sentence cut mean time to first audio byte from 329.3 ms to 156.7 ms, a 52.4 percent relative improvement, isolated to the TTS transport itself.

We then measured the fuller, end-to-end case with a real language model producing the text, running the identical prompt through run_bidi and run_batch against the same live Bedrock model and endpoint, so the only variable is when synthesis starts.

Using that same methodology, bidi mode’s time to first audio measured 2.4 times faster than batch mode against a local container, and 3.5 times faster against the deployed AgentCore runtime.

The following chart shows that gap across both environments:

Figure 4: Relative time to first audio for batch and bidi modes, locally and on the deployed runtime

The gap widens, rather than narrows, for the deployed case once a real network hop is added. run_bidi absorbs that hop inside the overlap it’s already exploiting between generation and synthesis. run_batch has no equivalent overlap to hide that cost in, since it does one thing at a time, so every added millisecond of network overhead lands directly on its serial wait.

To reproduce this comparison against your own deployment, run the same bidi-vs-batch test the agent container exposes:

uv run agent-infra demo --endpoint-url <your-endpoint-url> --prompt "<your prompt>"

This sends the prompt through both modes and reports the timestamp of the first audio_chunk event against agent_start for each. If you see errors instead, check for the KeyError: 'text' symptom from Step 1. An endpoint reporting InService while every request fails signals that bug, not a deployment issue.

Clean up

To avoid ongoing charges, tear down the resources this walkthrough creates in reverse deployment order, agent first, then endpoint:

uv run agent-infra destroy --bedrock-model-id <your-bedrock-model-id>
uv run speech-infra destroy kokoro-82m

agent-infra destroy removes the Amazon Bedrock AgentCore runtime resource. speech-infra destroy removes the SageMaker AI model, endpoint configuration, and endpoint. Neither empties the Amazon S3 bucket that caches model weights or deletes the Amazon ECR repository holding the container image, since both can be shared across models. If you no longer need them, empty and delete the Amazon S3 bucket and remove the Amazon ECR repository manually, since a stored model checkpoint and container image continue to incur small storage charges until removed.

Conclusion

We showed how to remove one source of avoidable latency from a voice agent, waiting for an agent’s full response before starting speech synthesis, and how to quantify a second, quieter cost: base64-encoded audio inside SSE inflates payload by about a third and can misalign with MP3 frame boundaries if you don’t buffer for it. Streaming each sentence into a SageMaker AI bidirectional endpoint as soon as an agent produces it cut time to first audio by more than half in isolation, and by 2.4 to 3.5 times in a deployed agent. Multiplexing binary audio and JSON control frames over that session, instead of encoding audio as text, removes the size inflation and frame-realignment work a naive SSE relay requires.

If you’re adding voice to an agent that already streams text over SSE, start with the base64 approach in Step 4 and revisit the unified transport once latency becomes the priority. To learn more, see the Amazon SageMaker AI real-time inference documentation and the Amazon Bedrock AgentCore documentation. If you’d like help applying this pattern to your own architecture, contact your AWS account team.

---

About the author