ElevenLabs Releases Eleven Realtime Real-Time Voice Model, Hands-On Test of Voice-Built Agent

On October 22, Zhidongxi reported that on October 20, the UK AI audio unicorn ElevenLabs released the Eleven Realtime series of real-time speech models, including Eleven Realtime v1 Mini for natural conversation and Eleven Realtime v1 Max for customer service and multi-step tasks. Both models support more than 90 languages and can use more than 17,000 voices from the ElevenLabs voice library.

This release focuses on simplifying the building and debugging of voice agents. In the past, developers had to separately select components such as speech recognition, large language models, and speech synthesis, and then adjust interaction details such as turn-taking and interruption. Eleven Realtime unifies the configuration and tuning of the underlying components; after developers choose Mini or Max, they can set roles, knowledge bases, and tools around specific business needs without having to select each underlying model one by one.

One of the difficulties of real-time speech is that it must both answer correctly and not make users wait too long. According to independent testing by third-party AI evaluation agency Artificial Analysis, Max ranked third in the Big Bench Audio speech reasoning benchmark with an accuracy of 98.8%, with an average time to first audio response of 0.57 seconds.

For this model release, Zhidongxi tested the Architect building assistant. From the hands-on experience, modifying Agent settings by voice is fairly convenient, and many operations do not require manual clicking. However, the Chinese accent is obvious, responses tend to be long, and latency is also relatively high.

ElevenLabs was founded in 2022 and started with text-to-speech, gradually expanding its business to speech recognition, sound effects, music, and conversational agents. In February this year, the company completed a $500 million (about 3.34 billion yuan) Series D funding round at a valuation of $11 billion (about 73.42 billion yuan); an employee share transaction in September brought its valuation to $22 billion (about 146.84 billion yuan). Investors include Sequoia Capital and a16z, as well as industrial capital such as Nvidia's NVentures and LG Technology Ventures.

Its two co-founders, Mati Staniszewski and Piotr Dabkowski, worked for many years at Palantir and Google, respectively, and in 2025 entered the ranks of Forbes billionaires. The company is also considering an IPO. CEO Mati Staniszewski previously said he hopes the company will be ready to go public within the next two to two and a half years.

Currently, the two models can be used through the ElevenAgents platform or the Realtime API. Users can create or open an Agent in the platform and select a Realtime model; developers can also integrate them into their own applications through the API. The API is compatible with the OpenAI Realtime specification and currently only supports WebSocket connections.

1. Hands-on test: modifying Agent settings by voice is convenient, but the Chinese accent is obvious

To test whether building an Agent by voice is convenient, Zhidongxi used ElevenLabs' Architect building assistant to try changing the language and switching voices in English, then asked the assistant to switch to Chinese communication and continued modifying settings in Chinese.

First, we asked in English to change the Agent's language setting. Architect understood the instruction and completed the modification; the process was fairly smooth and did not require manually finding menus or switching options.

Next, we asked it to change the voice. This time it did not work: Architect's own voice does not support switching, while the voice of the created Agent can be selected in the settings. After opening the voice library, it was clear that there are indeed many selectable voices.

▲Selectable voices in the ElevenLabs voice library

Then we asked Architect to switch to Chinese communication. It could understand the request, but the Chinese had an obvious foreign accent and sounded a bit artificial, with some expressions requiring careful listening to understand.

However, executing Chinese instructions went fairly smoothly. We used Chinese to request a change to the Agent's language setting, and it also completed the corresponding modification.

In the hands-on test, directly modifying settings by voice was still somewhat surprising. When users state their requirements clearly, the assistant can complete part of the configuration, saving the operations of finding menus and filling out forms.

However, Architect is somewhat verbose: even a simple request may trigger a long explanation, and it is not easy to interrupt.

▲When executing instructions, Architect sometimes gives longer explanations

In addition, during the test there was noticeable waiting for responses and occasional lag, possibly due to network effects. Long responses, insufficiently timely interruption responses, and Chinese speech that is not natural enough were the more prominent shortcomings in this experience.

Note: This hands-on test covered the Architect building assistant. Running the created Agent requires paid Tokens, so we were unable to continue testing the actual conversation performance of Mini and Max.

2. Mini is used for teaching and role interaction, Max handles customer service and multi-step tasks

In its official demo, ElevenLabs showcased the application scenarios of the two models.

The company says Mini focuses on low-cost natural conversation and is suitable for AI assistants, tutoring, and interactive characters; Max targets customer service, lead qualification, and multi-step workflows, making it better suited to tasks that require step-by-step reasoning and frequent tool calls.

Third-party AI evaluation firm Artificial Analysis independently tested both models. In the speech reasoning benchmark Big Bench Audio, Max scored 98.8% accuracy and Mini 75.9%. The test poses 1,000 questions by voice, covering tasks such as logical judgment, object counting, and path reasoning, and then scores the models' spoken answers.

As of October 21, Max ranked third on this individual item, behind StepFun's StepAudio 3 Realtime (99.7%) and Alibaba's Qwen Audio 3.0 Realtime Plus (99.2%).

▲Big Bench Audio speech reasoning benchmark ranking: Eleven Realtime v1 Max ranks third with 98.8% accuracy (Image source: Artificial Analysis)

Max also has lower latency. According to ElevenLabs' announcement, in the Big Bench Audio test the average time to first audio response for Max and Mini was 0.57 seconds and 0.62 seconds, respectively. For reference, the same metric for Qwen Audio 3.0 Realtime Plus listed on the Artificial Analysis website is 1.54 seconds. This metric measures the time a model needs to start speaking.

In the Speech Agent Arena evaluation, in which real people take part in real-time conversations, Max achieved an 87.1% tool-use task completion rate and Mini 60.3%. This metric measures the proportion of tasks a model completes through correct tool calls; Max is 26.8% higher than Mini on this item.

However, on turn-taking behaviors such as pauses and interruptions, Mini's overall score was slightly higher. In Full-Duplex-Bench (a full-duplex dialogue evaluation), Artificial Analysis selected four categories—pause handling, turn handoff, user interruption, and backchannel responses—and calculated a weighted score: Mini scored 95.8% and Max 94.1%. These items examine whether a model can keep waiting when the user pauses, respond promptly after the user finishes speaking, and distinguish genuine interruptions from backchannels such as "mm" and "right."

3. From dubbing and transcription to real-time interaction, continuing the cascaded architecture

Before this Realtime launch, ElevenLabs had already released speech synthesis and speech recognition models and offered a voice Agent building service.

On speech synthesis, in September this year the company released the text-to-speech model Eleven v4 and its low-latency version v4 Turbo, focusing on improving emotional expression, voice cloning, and multi-character dialogue. Users can control the tone and stress of lines with natural language or tags and add sounds such as laughter, while v4 Turbo is optimized for low-latency scenarios such as voice Agents.

On speech recognition, Scribe v2 Realtime, launched last November, targets real-time transcription; Scribe v2, released this January, focuses on long recordings and complex audio, supporting speaker diarization, word-level timestamps, and hotword prompting.

Beyond individual models, ElevenLabs launched Conversational AI as early as November 2024, enabling developers to create voice Agents by combining knowledge bases, functions, and large language models.

On the technical route, Eleven Realtime continues to use a cascaded architecture, in which independent components such as speech recognition, large language models, and speech synthesis work together to complete a conversation. The company believes this approach makes it easy to test and upgrade each stage separately, and also allows errors to be diagnosed and responses constrained through the intermediate text, as well as integration of stronger large language models.

However, the company also notes in a technical blog that converting speech to text loses some intonation, rhythm, and emotional information, which must be supplemented through additional modeling. Its optimization direction is to improve voice performance, latency, and turn handoff while preserving the controllability of each component.

On pricing, the Realtime API is billed by audio connection duration and Token usage. Beyond the plan quota, the audio connection fee for both models is $0.08 per minute; Mini charges $0.20 per million input Tokens and $6 per million output Tokens; Max charges $1.80 and $9, respectively.

Conclusion: Real-time voice is accelerating into Agents, but the interaction experience still needs polishing

With this Realtime launch, ElevenLabs combines speech recognition, reasoning, and speech generation into a ready-to-use real-time interaction solution. While improving conversation performance, it also lowers the barrier to building voice Agents.

This track already gathers several major players. OpenAI's GPT-Realtime, Google's Gemini Live, and Alibaba's Qwen-Audio-Realtime series are all strengthening real-time voice interaction and tool-calling capabilities. Although their technical routes and product emphases differ, their goal is the same: to let users drive AI to complete specific tasks through natural conversation.

To make conversations smooth and keep tasks moving forward, a complete system is also needed. Network latency, turn-taking timing, and tool execution all affect the user experience; OpenAI has also disclosed optimizations to its underlying network architecture to reduce pause and interruption latency.

From Zhidongxi's hands-on experience this time, creating documents and modifying settings by voice was already surprising, but response speed, interruption handling, and voice continuity during page switches still need polishing. As models and products continue to iterate, we expect such applications to perform more stably in continuous conversation and task execution, making voice a truly handy way for users to operate in their daily work.