FireRedTeam 开源 FireRedAudio 通用音频语言模型

FireRedTeam; FireRedAudio

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork20 (20); Star2k (2k)You must be signed in to star a repository

Repository files navigation

One model to listen, understand, reason, speak, and edit.

FireRedAudio is a general-purpose audio language model built on a shared 9B-parameter LLM with decoupled continuous representations: an Audio Encoder handles understanding, while a RedAE pathway handles generation. A single model supports ASR, audio understanding, zero-shot TTS, instruct TTS, semantic/acoustic speech editing, and accurate temporal grounding over recordings up to one hour long.

Highlights ✨

🧩 Purpose-built representations, one shared backbone — The Audio Encoder pathway serves understanding, while the RedAE-Patch pathway serves speech generation. Their representations remain decoupled but share the same language and reasoning backbone. To the best of our knowledge, this is the first publicly disclosed design of its kind in a unified audio-language model; 📊 One model, a full audio stack — FireRedAudio spans ASR, broad and fine-grained audio understanding, zero-shot TTS, Instruct TTS, and free-form speech editing, achieving competitive or leading results across MMAU, MMSU, Seed-TTS-Eval, InstructTTSEval, and Ming-Freeform-Audio-Edit; 🎙️ Create and edit speech with natural language — Clone a voice from a reference clip, design a voice from a description, or edit what was said and how it sounds through one continuous-latent generation pathway; ⏱️ Go from minutes to hour-long recordings — Understand recordings up to one hour with precise time-to-content alignment. Organize audio into timestamped structures, produce grounded summaries, retrieve content by time (or time by content), and reason over evidence distributed across the recording

[2026.09.08] We release a Gradio web demo for zero-shot TTS, instruct TTS, speech editing, ASR, and audio understanding; [2026.08.21] We release the FireRedAudio code and model

Requires Python 3.10 and uv. System prerequisites: a CUDA toolkit (GPU inference + compiling the causal-conv1d / flash-attn kernels) and ffmpeg (audio decoding via torchaudio/torchcodec).

Wheels target CUDA 12.8 (cu128) by default. For a different CUDA version, change the pytorch-cu128 index URL in pyproject.toml (e.g. cu126 / cu129) and re-run uv sync.

Model Download

Download the pretrained model from Hugging Face with the hf CLI:

Alternatively, download the pretrained model using modelscope CLI:

A single-file Gradio app at app.py wraps all four tasks (zero-shot TTS · instruct TTS · speech editing · listen & understand) with the same weights and prompt templates as the CLI. Gradio is the only extra dep on top of the base install:

Defaults to http://0.0.0.0:7860. The understanding tab exposes the full Qwen3 sampling controls (temperature, top_p, top_k, min_p, repetition_penalty, do_sample) with a think-preset toggle that swaps in Qwen3's recommended (temperature, top_p) for reasoning / non-reasoning modes. If you're serving behind a reverse proxy, add --root-path .

Performance

Audio Understanding

Results marked with * are obtained from our own evaluation

ASR

Results marked with * are obtained from our own evaluation

Zero-Shot TTS

Instruct TTS

These results are obtained from our own evaluation

Third-party ports and integrations of FireRedAudio, maintained by the community.

Limitations

Everything but ASR is Chinese/English only. Speech generation (tts / edit / voice_design) and audio understanding are limited to Chinese and English — tts selects the language via --language zh / en. ASR is the only task that supports more languages; Zero-shot TTS is not deterministic by default. The flow-matching decoder samples random noise, so output varies run to run; pass a fixed seed (set_seed(...) in the API, --seed on the CLI) for reproducibility, and note that quality can differ across seeds; Long-form input is supported up to about one hour. Beyond that the model is untested and time-to-content alignment may degrade

Usage Disclaimer

The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended solely for academic research purposes; DO NOT use this model for ANY illegal activities❗️❗️; The developers assume no liability for any misuse of this model; If you identify any instances of abuse, misuse, or fraudulent activities related to this project, please report them to our team immediately

Citation

Acknowledgements

Qwen3.5 for the language model foundation; Whisper-large-v3 for the Audio Encoder initialization; x-transformers for RotaryEmbedding; vocos for ISTFT implementation

License

Released under the Apache-2.0 license.