FireRedTTS3 发布：支持 24 种语言与 21 种中文方言的语音克隆

FireRedTeam; FireRedTTS3

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork21 (21); Star1.8k (1.8k)You must be signed in to star a repository

Repository files navigation

FireRedTTS3 is a unified speech generation and editing system built on semantically enriched continuous speech representations. It comes in two variants:

FireRedTTS3-Base — zero-shot voice cloning across 24 languages and 21 Chinese dialects; FireRedTTS3-Instruct — natural-language voice design and speech editing (semantic + acoustic) in one unified model

Highlights ✨

🌍 Multilingual — 24 Languages — Best average WER/CER (avg 3.754%) and best average speaker similarity on MiniMax-MLS-Test (avg 84.8%), plus best-in-class cloning WER/CER (avg 3.04%) and similarity on Seed-TTS-eval (avg 78.8%). Supported languages: Arabic · Cantonese · Chinese · Czech · Dutch · English · Finnish · French · German · Greek · Hindi · Indonesian · Italian · Japanese · Korean · Polish · Portuguese · Romanian · Russian · Spanish · Thai · Turkish · Ukrainian · Vietnamese; 🗣️ Multi-Dialect — 21 Chinese Dialects — Zero-shot voice cloning across major Chinese dialect groups. Supported dialects: Anhui · Fujian · Gansu · Guizhou · Hebei · Henan · Hubei · Hunan · Jiangxi · Liaoning · Minnan · Ningxia · Shaanxi · Shandong · Shanghai · Shanxi · Sichuan · Tianjin · Wenzhou · Wu · Yunnan; 🎨 Instruction-Controlled Voice Design — Generate a brand-new voice from a natural-language description (gender, age, timbre, emotion, pace, accent…) with no reference audio, guided by an explicit textual plainning step before synthesis; ✂️ Free-Form Speech Editing — Semantic editing (insertion / deletion / substitution) and acoustic editing (speed / pitch / volume) driven by free-form instructions

[2026.08.13] We release the Gradio demo for both FireRedTTS3 Base & Instruct; [2026.08.13] We release the FireRedTTS3

Roadmap

Release the FireRedTTS3-Base model; Release the FireRedTTS3-Instruct model; Release the technical report

Contents

Download the pretrained model from Hugging Face with the hf CLI:

Alternatively, you can download model from ModelScope

Configure Text Frontend

Language Recognition (Optional)

FireRedTTS3-Base relies on explicit language tags for best performance. However, if you don't know the exact language of the text, you can download Meta's FastText language-id model and let it detect the language automatically.

Text Normalization (TN)

TN converts written numbers, dates, units, currencies, acronyms, etc. into their spoken form (e.g. 19:30 → nineteen thirty). By default, FireRedTTS3 uses the wetext TN tool, which supports Chinese and English, other languages (e.g. Japanese, Russian) undergo only basic cleaning. For full language TN support, enable the LLM-based TN by passing use_llm_tn=True when initializing FireRedTTS3. It reads its config from a .env file:

Python API

For the best voice cloning performance, use a prompt in the desired language or dialect, since the output inherits the speaking style of the reference. For example, provide a Japanese prompt when synthesizing Japanese and a Sichuanese prompt when synthesizing Sichuanese.

FireRedTTS3-Instruct is a unified instruction-driven model. On top of zero-shot voice cloning, it also supports Voice Design, Semantic Edit and Acoustic Edit through a single entry point: fireredtts3.core.FireRedTTS3Instruct.

Gradio Demo

Gradio interface for FireRedTTS3-Base(left) and Instruct(right).

Performance

Best in bold, second best in underline. Evaluation scripts: Seed-TTS-eval.

Best in bold, second best in underline. CER reported for Chinese, Cantonese, Japanese, Korean, Arabic, Vietnamese, Hindi, Thai, and Greek; WER for the rest.

Instruct TTS

Since Gemini-2.5-pro-preview is inaccessible, Gemini-2.5-pro is used to score all systems.

Speech Editing

Usage Disclaimer

The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended solely for academic research purposes; DO NOT use this model for ANY illegal activities❗️❗️; The developers assume no liability for any misuse of this model; If you identify any instances of abuse, misuse, or fraudulent activities related to this project, please report them to our team immediately

Citation

Acknowledgements

Qwen3 and Qwen2-Audio for the language model and audio understanding foundations; DiTAR for the patch-level diffusion autoregressive formulation; X-Codec for the discriminator design used in RedAE training; CAM++ for speaker embedding extraction; fastText for automatic language identification; WeTextProcessing (wetext) for the Chinese / English text normalization front-end

License

Released under the Apache-2.0 license.