FireRedTTS2 开源长时流式多说话人对话 TTS 系统

FireRedTeam; FireRedTTS2

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork131 (131); Star1.4k (1.4k)You must be signed in to star a repository

Long-form streaming TTS system for multi-speaker dialogue generation

Repository files navigation

FireRedTTS‑2 is a long-form streaming TTS system for multi-speaker dialogue generation, delivering stable, natural speech with reliable speaker switching and context-aware prosody.

Highlight🔥

Long Conversational Speech Generation: It currently supports 3 minutes dialogues with 4 speakers and can be easily scaled to longer conversations with more speakers by extending training corpus; Multilingual Support: It supports multiple languages including English, Chinese, Japanese, Korean, French, German, and Russian. Support zero-shot voice cloning for cross-lingual and code-switching scenarios; Ultra-Low Latency: Building on the new 12.5Hz streaming speech tokenizer, we employ a dual-transformer architecture that operates on a text–speech interleaved sequence, enabling flexible sentence-bysentence generation and reducing first-packet latency，Specifically, on an L20 GPU, our first-packet latency as low as 140ms while maintaining high-quality audio output; Strong Stability：Our model achieves high similarity and low WER/CER in both monologue and dialogue tests; Random Timbre Generation:Useful for creating ASR/speech interaction data

Demo Examples

Random Timbre Generation & Multilingual Support

Zero-Shot Podcast Generation

Speaker-Specific Finetuned Podcast Generation

⚠️ Speaker voices: hosts "肥杰" and "惠子" from the podcast "肥话连篇". Use without authorization is forbidden.

⚠️ 声音来源：播客 "肥话连篇" 主播 "肥杰" 和 "惠子"，未经授权不能使用。

For more examples, see demo page.

[2025/10/26] 🔥 We have released comprehensive fine-tuning code and tutorials (based on the LJSpeech dataset). You can easily adapt this foundation to fine-tune models for multilingual or even conversational datasets. With conversational data, you can achieve podcast generation results comparable to our demo showcase; [2025/10/11] We now support streaming dialogue generation; [2025/09/28] Supports bf16 inference, reducing VRAM usage from 14GB to 9GB and enabling consumer-grade GPU deployment; [2025/09/12] We have added a UI tool to the dialogue generation; [2025/09/08] We release the pre-trained checkpoints and inference code; [2025/09/02] We release the technical report and demo page

Roadmap

2025/09

Release the pre-trained checkpoints and inference code; Add web UI tool; 2025/10

Release a base model with enhanced multilingual support; Provide fine-tuning code & tutorial for specific dialogue/multilingual data; End-to-end text-to-podcast pipeline

Install & Model Download

Clone and install

Clone the repo; Create env: Setup environment with Conda:

or use Docker:; Model download

Basic Usage

Dialogue Generation with Web UI

Generate dialogue through an easy-to-use web interface that supports both voice cloning and randomized voices.

Dialogue Generation

Dialogue Generation (Sreaming)

NOTE: Each audio chunk is 0.08 seconds, except the first (a little shorter) and last (a little longer).

Monologue Generation

Acknowledgements

We thank Moshi and Sesame CSM for their novel dual-transformer approach. Additionally, we adapted Sesame CSM's structure and core inference code; We referred to Qwen2.5-1.5B text tokenizer solution; We referred to Xcodec2 Vocos-based acoustic decoder

⚠️ Usage Disclaimer ❗️❗️❗️❗️❗️❗️

The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended solely for academic research purposes; DO NOT use this model for ANY illegal activities❗️❗️❗️❗️❗️❗️; The developers assume no liability for any misuse of this model; If you identify any instances of abuse, misuse, or fraudulent activities related to this project, please report them to our team immediately

Star History

Long-form streaming TTS system for multi-speaker dialogue generation