Voice Arena launches the Turing TTS benchmark: the best model fools listeners only about 10% of the time
Voice Arena has launched the Turing TTS benchmark, in which listeners choose the real human among a real human recording and a model-generated recording of the same sentence; the proportion of wrong choices is the Fool Rate, and 50% means indistinguishable. The best-performing Gemini 3.8 Flash TTS made listeners choose wrongly in about 10% of cases. On Voice Arena's preference leaderboard, the top four models are statistically tied, but overall they still show a clear gap relative to real humans.
Why read this
AI score 59
The new benchmark breaks speech naturalness down into a quantifiable Fool Rate and gives the specific gap between each model and real humans in terms of being hard to tell apart.
Topics
View original postDrop 2/15 Are we actually measuring the things we ultimately want our voice AI models to achieve? The best AI voices are becoming hard to tell apart. From each other. Not from humans. 🚀 Introducing Turing TTS: an evaluation that measures how often human listeners mistake AI-generated speech for real human speech. We tested leading TTS models against real human speech. Even the best model fooled listeners only ~10% of the time. With Turing TTS, we hope to contribute to a broader discussion within the research community about how we measure progress in voice AI. If one of our goals is to develop speech that is indistinguishable from human speech, how well do our current evaluation methods capture progress toward that goal? We don’t claim that Turing TTS is the right way forward, or that human indistinguishability should be the goal of every TTS system. Rather, we hope these findings encourage the community to collectively examine whether blind pairwise human evaluations of naturalness are sufficient, and what else we might need to measure. Results are now live on @voicearena_ai.
View original
Drop 2/15 Are we actually measuring the things we ultimately want our voice AI models to achieve? The best AI voices are becoming hard to tell apart. From each other. Not from humans. 🚀 Introducing Turing TTS: an evaluation that measures how often human listeners mistake AI-generated speech for real human speech. We tested leading TTS models against real human speech. Even the best model fooled listeners only ~10% of the time. With Turing TTS, we hope to contribute to a broader discussion within the research community about how we measure progress in voice AI. If one of our goals is to develop speech that is indistinguishable from human speech, how well do our current evaluation methods capture progress toward that goal? We don’t claim that Turing TTS is the right way forward, or that human indistinguishability should be the goal of every TTS system. Rather, we hope these findings encourage the community to collectively examine whether blind pairwise human evaluations of naturalness are sufficient, and what else we might need to measure. Results are now live on @voicearena_ai.
Body
Voice AI is now good enough that preference and human-likeness can be scored as separate targets.
@voicearena_ai launched a new benchmark, Turing TTS to score the second one directly: listeners hear the same sentence from a person and from a model, then pick the person.
How often they pick the model is the Fool Rate. 50% would mean listeners can’t tell.
Even the best model, Gemini 3.8 Flash TTS, fooled them about 10% of the time.
On Voice Arena’s preference board, the top four models are statistically tied. Measured against a human, the whole field still has a long way to go.
https://t.co/AUS2S4Wp5h