Armin Ronacher's Notes on Custom Voices with Qwen3-TTS

Armin Ronacher's Notes on Custom Voices with Qwen3-TTS

When Armin plays a little dungeon-crawler game with his kid, he needs narration but doesn't want a generic AI voice, so he thought of using Qwen3-TTS's Base to clone a speaker embedding of about 2048 dimensions from a sample; by slotting it into the same speaker slot in CustomVoice, he can add "sad / whisper"-style instructions on top of the cloned voice; the embedding can also be weight-blended, baked back into a reference audio, or even shifted along a "average female voice − average male voice" direction to move the perceived gender, making it highly playable! https://t.co/N3EiKHykv3

Qwen3's three model types · Base (1.7B / 0.6B): clone from a few seconds of reference audio; supports in-context cloning or using only the speaker embedding · CustomVoice: nine built-in speakers + instructions (such as emotion, style) · VoiceDesign: "invent" a voice from a text description

The problem Armin ran into: Base can clone, but along the documented path you can't clone and give instructions ("say it sadly") at the same time. The official workarounds are mostly: fine-tune Base on a single speaker, or generate a segment with VoiceDesign and then clone it with Base, which loses the instruction capability.

How did Armin do it? He cut about 28 seconds of complete sentences between pauses from a PyAI Conf talk and ran Whisper to get a transcript. He stresses: it doesn't need to be especially clean or especially long; in a low-quality phone-call scenario, fooling acquaintances isn't hard (a reminder of the risk of abuse).

You can use VoiceDesign to first create a voice that is "around forty, warm, low-to-mid pitch, reading calmly," then compare and blend it with the person's own recording.

Embeddings and blending In x_vector_only_mode, Base extracts a speaker embedding (about 2048 numbers) from the recording. The same person's embeddings are close, different people's are apart; it can also roughly do speaker segmentation. Because it's a vector, you can do a weighted average and scale by norm to get a voice "between two people." In the post he blends his own voice with the designed voice in eleven steps: it quickly stops sounding like him, but still retains some features and is overall more distinctive. It works better when the two recordings have similar spectra (similar mic and environment). A blend you like can be re-synthesized into an audio clip, re-fingerprinted, and baked back into a single reference file (not exactly equal to the original blend, but easier to manage).

On speaker recognition: using a meeting clip as reference to recognize himself in a podcast, Qwen3 almost can't do it; switching to about 20 seconds of self-introduction at the start of the podcast as reference, it gets about 98% right on subsequent windows. A dedicated speaker-verification model (such as ECAPA) can also recover about 97% using the meeting clip.

Giving instructions to a cloned voice Observation: CustomVoice and Base manipulate the speaker in the same kind of way, inserting a vector into the talker input sequence; CustomVoice uses a row from a table, Base uses an x-vector; the generation code can take either, and the weights are almost the same. So: extract the embedding with Base, hand it to CustomVoice's voice_clone_prompt, then add instruct. Instructions like sadness are less stable on a cloned voice than on built-in speakers, but Armin feels it's already enough for custom game narration that "won't collide with what's in other people's games."

Gender-axis arithmetic VoiceDesign makes 14 male / 14 female voices each, takes the average difference as a direction, and shifts any embedding along that axis (keeping the remaining components). The post shows examples of his own clone moving along the axis. Closing rem: "Use it for good, not evil."