Pydantic AI 上线实时语音智能体

A Pydantic AI agent is a plain Python object with no interface baked in. The same agent already runs as an object you call run() on, in the terminal, behind a built-in web chat, and streamed to your own frontend. Now it can hold a live, spoken conversation too. Voice is just another interface on the agent you already have.

Under the hood it's realtime speech-to-speech: OpenAI Realtime, Azure OpenAI, Gemini Live, and xAI Grok Voice, behind one provider-agnostic API. The model hears and speaks directly, with no transcribe-then-generate-then-synthesize pipeline in between, so latency is low and interruptions feel natural.

Your backend always runs the agent, so tools, history, and your API key stay server-side. The audio travels however suits your app: captured on the server, or, for a browser, directly between the browser and the provider over WebRTC, with your backend attached to the same call as a sideband so nothing moves client-side but the audio. Browser WebRTC works on OpenAI and Azure OpenAI, with a runnable FastAPI example to start from.

#A voice agent in one file

Realtime lives behind an extra. Install Logfire next to it and the session traces itself:

With the audio captured on the server, a voice agent is one session and three small loops (microphone in, speaker out, and a live transcript):

send_audio() streams the caller's microphone in, stream_audio() streams the spoken reply back, and stream_transcripts() gives you a live transcript of both sides. The voice assistant example fills the two audio placeholders in with sounddevice. That agent is not a new VoiceAgent type: it's the same Agent, and get_weather is your ordinary server-side tool, called mid-call. The two logfire lines are the whole of the observability setup.

Building agents in production?

See what they're actually doing with Logfire — traces, evals, cost, and behavior in one view.

#What carries over

Opening a realtime socket to a provider is a few lines in any SDK. The work is everything around the model: tools that actually run, history you can trust, costs you can see, and a security model that keeps your API key off the browser. That's the part Pydantic AI does, and it works the way it does for text agents:

Your typed tools run server-side. The same tools you registered with @agent.tool, with their dependencies, validation, and retries. Each call runs in the background so it never blocks the session; whether the model keeps speaking while it waits is provider-specific; Your capabilities carry over. The same opt-in behaviors you compose onto a text agent, including third-party ones, resolve once at connect time. (Some run-graph features like output validators don't apply.); The session builds real message history. Spoken turns become the same ModelRequest/ModelResponse messages a text run produces, including tool calls, and transcripts when the provider supplies them; Usage and limits work. session.usage accumulates tokens with audio and cached breakdowns; usage_limits caps a runaway session the same way it caps a run; It's instrumented. A realtime session emits OpenTelemetry spans the way an agent run does, so Logfire or any other OTel backend reads it without special handling

#One agent, many modalities

Because a voice session records canonical message history, voice and text compose. Hand a finished call to a text agent for structured extraction:

The voice call and the text summary are two agents sharing one message history. It goes the other way too: seed a voice session with message_history= from an earlier text conversation and the caller picks up where they left off. The phone bot and the assistant in your app can be the same Agent with the same audit trail.

#Portable across providers, and through the gateway

Every provider implements the same RealtimeModel interface and normalizes into one typed event vocabulary, so the core of your event loop stays the same when your provider changes. Each model declares its capabilities on its profile: manual turn-taking, barge-in truncation, non-blocking tool calls, and the provider-native tools it supports, like Gemini's search grounding. You branch on what a model can do instead of discovering it mid-call.

Portability includes the Pydantic AI Gateway. Route a session through it by naming the upstream provider, and nothing else about your code changes:

Your voice traffic then gets the same single key, spend limits, and routing as your text traffic. Apart from obvious reasons, this matters for audio because realtime usage is billed by each provider's audio-token or per-minute pricing, so an unattended call can run up cost very fast.

#Try it

Realtime support ships in pydantic-ai today:

That covers OpenAI, Azure OpenAI, and Gemini Live, because the pydantic-ai package already bundles those SDKs. Add [xai-realtime] for xAI Grok Voice. On the lean pydantic-ai-slim package, name the provider yourself: pydantic-ai-slim[openai-realtime] (also covers Azure), [google-realtime], or [xai-realtime].

Start with the realtime docs, then pick an example: a terminal voice assistant, the browser WebRTC app, a camera agent that watches and narrates, or handing a call off to a text agent.

#See what the call did

Voice arrives in the trace as text. Spoken turns land under pydantic_ai.all_messages, the attribute a text run already writes, with transcripts standing in for the audio.

One session span holds the whole call, with a user speech span for each stretch the caller talked and a chat span for each reply. A tool call mid-conversation gets its own span, with the arguments the model chose and what your function returned. pydantic_ai.audio_chunks_dropped and transcript_items_dropped count what the session could not keep up with.

Below is the agent trace from this post on a real call: four questions, three execute_tool get_weather spans, 61.87s costing $0.08, or about $4.50 for an hour of talking at that rate.

Pydantic Logfire files each session alongside your other agent runs, since the session span carries the same agent_name its Runs view groups on.

If you build something with it, or hit something that doesn't feel right, we want to hear about it, on GitHub or in Slack.