CUHK-Shenzhen and Microsoft Research Propose ConvexLLM, a Unified Multimodal Modeling Architecture
The Chinese University of Hong Kong, Shenzhen and Microsoft Research jointly proposed ConvexLLM, which uses C-Gate to map audio representations into the convex hull of a large language model’s text token embeddings, thereby drawing on the model’s existing knowledge and reasoning capabilities. ConvexLLM scored 75.36 across 11 VoiceBench tasks and achieved 90.33% and 80.40% on OpenBookQA and BBH, respectively. In a same-backbone comparison using Qwen3.5-2B, C-Gate achieved an overall speech score of 60.35, outperforming MLP at 58.80 and Q-Former at 54.19.
Why read this
AI score 51
The study constrains audio representations within the convex hull of text token embeddings, providing a concrete approach for examining how speech interfaces inherit language models’ knowledge and reasoning capabilities.
Topics
Body · AI translation
New public beta promotional copy New public beta promotional copyend Add app-exclusive traffic redirection
This study was jointly conducted by The Chinese University of Hong Kong, Shenzhen and Microsoft Research. Corresponding author Zhizheng Wu is an associate professor at The Chinese University of Hong Kong, Shenzhen, a recipient of a national-level young talent award, and the founder of Amphion Technology. Shujie Liu is a principal researcher at Microsoft Research who has long worked on natural language processing and speech processing. Minghao Xu, Yuxiang Wang, and Yingda Shen are doctoral and master’s students at CUHK-Shenzhen. Yuxuan Hu and Jinyu Li are from Microsoft CoreAI. The newly proposed ConvexLLM uses the innovative C-Gate architecture to map speech information into the convex hull of a large language model’s existing text token embeddings, allowing the model to fully leverage its accumulated knowledge and reasoning abilities. Across multiple benchmarks, including VoiceBench, ConvexLLM delivered leading performance among open-source models of a comparable scale, providing a new technical path toward unified speech and language modeling.
Research institutions: School of Data Science, The Chinese University of Hong Kong, Shenzhen; Microsoft CoreAI; Microsoft Research Asia
Large models have already learned knowledge and reasoning. If we give one a pair of “ears,” can those capabilities carry over as well?
This is an unavoidable question for speech large language models: the same question may be answered correctly when written down, yet performance can decline significantly when it is spoken. The sound has already been encoded into vectors, and the language model is powerful enough—so what is still missing between the two?
A research team from The Chinese University of Hong Kong, Shenzhen and Microsoft turned its attention to the interface through which sound enters a language model. ConvexLLM constrains audio representations to the convex hull of the language model’s existing text token embeddings, allowing sound to enter the large model as continuous combinations of text vectors.
ConvexLLM Demo
What is a “convex hull”? Imagine several nails driven into a tabletop and a rubber band stretched around them from the outside and pulled tight. The region enclosed by the rubber band, including its boundary, is the convex hull of those points. If the points are combined using a weighted average with nonnegative proportions totaling 100%, the resulting point will always lie within this region. Text token embeddings can likewise be viewed as points in space, only in far more dimensions; ConvexLLM represents sound by mixing these token embeddings.
This design, which appears to impose an additional constraint, delivers striking performance in speech-based knowledge and reasoning.
In the paper’s comparison of similarly scaled open-weight models, ConvexLLM ranked first with an aggregate score of 75.36 across 11 VoiceBench tasks. It achieved 90.33% on OpenBookQA and 80.40% on BBH, with the latter exceeding the strongest baseline in the table by 8.80 percentage points.
This research brings to the fore a question worth pursuing across the entire multimodal field: just how much rich audio information can the continuous space formed by text token embeddings carry?
I. Letting sound use coordinates familiar to the language model
Through pretraining, text-based large language models accumulate extensive knowledge, reasoning patterns, and instruction-following abilities. A central goal in building speech large language models is to enable users to invoke these abilities simply by speaking.
After sound passes through an encoder, an interface must convert it into input for the language model. Common approaches such as MLPs or Q-Formers can map it freely within a continuous space. ConvexLLM chooses a clear reference for this step: the language model’s own text token embeddings.
Its core interface, C-Gate, represents each audio position as a weighted combination of multiple frozen token embeddings. The token embeddings provide stable coordinates, while the sound determines how they are combined. This design grounds speech adaptation in representations the language model already possesses.
From this perspective, for speech and text expressing the same content, differences between modalities do not necessarily mean that the semantics themselves differ; they may instead manifest as different trajectories through a shared representation space. Text organizes information through discrete tokens, whereas speech can carry content and acoustic details through combinations of continuous vectors that change over time. C-Gate provides a common coordinate basis for exploring this possibility.
Let sound use representations familiar to the large model and tap into the intelligence it has already learned.
II. The token embeddings are fixed; the combinations are flexible
The most interesting aspect of “placing sound in text space” is continuous combination.
The language model’s token embeddings can be thought of as a palette. The pigments are fixed, but changing their proportions can produce a rich range of colors. C-Gate does something similar with sound: each audio position first forms a query, selects the 16 best-matching token embeddings from the vocabulary, and then learns the weights used to mix them.
These weights are nonnegative and sum to 1, so the mixed representation lies within the convex hull of the text token embeddings. As the sound changes, the selected vectors and their mixing proportions change with it, and the representations at successive audio positions form a temporal trajectory within the convex hull of the text token embeddings. The reference frame of the representation space remains fixed; what changes is the path traced by the sound within it and the information carried along that path. Figure 1 | C-Gate continuously mixes the selected Top-16 frozen token embeddings, allowing sound to enter the language model.
This preserves a crucial degree of expressive freedom: an audio representation can lie between multiple token embeddings, without corresponding to any single discrete token or first being converted into a readable transcript. What the language model receives remains a sequence of continuous representations determined by the sound.
The importance of continuous mixing is especially clear in the support-set-size experiment: under the paper’s 2B configuration, when only one token embedding is selected at each position, the overall VoiceBench score is 22.30. After expanding to a continuous combination of 16 token embeddings, the overall score reaches 60.35. This freedom of combination makes the fixed token-embedding space more expressive.
Training is also carried out progressively around this interface: first the interface is trained, then the encoder is added, and finally the encoder, interface, and decoder modules are jointly adapted. The text token-embedding table remains fixed throughout. The main model also uses text-teacher distillation, enabling the response distribution for speech input to acquire the language capabilities demonstrated with the corresponding text input.
III. A Score of 75.36: Connecting Sound to Stronger Knowledge and Reasoning
The value of an interface design ultimately depends on how much capability it brings to the speech side.
On VoiceBench, which covers tasks including spoken interaction, question answering, instruction following, and reasoning, ConvexLLM achieves an overall score of 75.36 across 11 tasks, outperforming MiniCPM-o 4.5 at 73.04, Kimi-Audio at 70.26, and MiMo-Audio at 67.01. Figure 2 | ConvexLLM leads the overall rankings across 11 VoiceBench tasks with a score of 75.36.
The most striking results are concentrated in knowledge and reasoning. ConvexLLM achieves the highest score in the table on all four tasks—OpenBookQA, MMSU, SpeechMMLU, and BBH. Figure 3 | ConvexLLM surpasses the strongest respective baseline on all four knowledge and reasoning tasks, leading BBH by 8.80 percentage points.
In particular, BBH rises from the strongest baseline’s 71.60% to 80.40%, while OpenBookQA improves from 83.70% to 90.33%. These tasks require the model to draw on knowledge, analyze conditions, and complete reasoning after listening to a question. The gains occur in precisely the capability we most want speech LLMs to acquire: not merely understanding what they hear, but genuinely thinking the problem through.
For application research in areas such as voice assistants, spoken-language learning, knowledge-based question answering, and professional consulting, this offers a highly attractive direction: enabling the speech interface to connect more effectively to the reasoning capabilities already possessed by the language model.
IV. The Convex Hull of Text Can Also Carry Environmental Sounds and Music
ConvexLLM further applies the same interface to understanding speech, environmental sounds, and music.
On MMAU Test-mini, the model achieves an overall accuracy of 62.50%, including 65.77% for environmental sounds, 60.78% for music, and 60.96% for speech. It also performs speech recognition, achieving a word error rate of 2.39% on LibriSpeech test-clean.
This gives the “text space” a more imaginative meaning: token embeddings originally intended for text can be continuously combined to carry the information required by multiple audio tasks. The vocabulary is finite, but its combinations preserve a rich space of variation for audio.
Using existing acoustic encoders and language models, ConvexLLM uses at most 271,000 hours of audio during adaptation. A clearly structured interface is already sufficient to support a complete research system spanning speech recognition, knowledge-based reasoning, and non-speech audio understanding.
V. The Interface Itself Is a Variable Worth Studying Seriously
If you are working on a speech LLM, what you will probably want to know next is how much difference the choice of interface itself can make.
Using Qwen3.5-2B, the research team compared MLP, Q-Former, and C-Gate interfaces, each with approximately 1.7 million parameters. All three use the same encoder and decoder and share the same Stage 3 data, training objective, and optimization settings. They were evaluated after 18,000 steps of Stage 3 training. Figure 4 | With the same 2B backbone, C-Gate achieves the highest overall speech score and the smallest gap from the corresponding text input.
C-Gate achieves an overall speech score of 60.35, higher than MLP’s 58.80 and Q-Former’s 54.19. The overall score with text input is 63.20. Among the three interfaces, C-Gate also has the smallest overall-score gap between speech and the corresponding text.
Representation analysis provides a more intuitive clue. In another set of 2B diagnostic models, by the seventh decoder module, the centered cosine similarity between C-Gate’s speech representations and the corresponding text representations has already reached 0.69, compared with 0.45 for MLP and 0.44 for Q-Former. At a relatively early stage of the model’s computation, speech begins to follow a representational path closer to that of text.
This gives interface research a very concrete point of inquiry: where does sound enter the language model, and how does that position affect subsequent knowledge retrieval and reasoning? ConvexLLM turns this question into an experimental design that can be implemented, compared, and analyzed.
VI. Your Next Study Could Begin with This Interface
The aspect of ConvexLLM most worthy of further exploration is that it provides a simple, clearly defined starting point from which research questions can readily emerge.
For speech LLM research, start with stronger capability transfer
Include C-Gate as a bridge baseline and use the same backbone to study stronger spoken-instruction following, complex question answering, and reasoning. The interface gives sound a stable textual reference frame, making it possible to continue exploring how distillation signals, data recipes, and training stages can work together to bring more language capabilities to the speech side.
For representation learning, directly ask what the text space can contain
Content, emotion, speaker identity, temporal order, and environmental events can each be studied through the same interface. By changing the number of mixed token embeddings, the degree of input compression, or the backbone model, researchers can observe which information enters the representation and which information is actually used by the decoder. How rich a world of sound can be composed from a finite set of token embeddings? This is a question that is both specific enough and large enough to matter.
For multimodal adaptation, use it as a preliminary study
ConvexLLM’s interface concept provides a clear experimental starting point for further exploration: fix the token-embedding table, learn how to combine embeddings for the input, and then compare different encoders, fixed or trainable decoders, and new modalities and tasks. It turns “what existing language intelligence can accommodate” into a series of design choices that can be tested step by step.
For researchers, the value is straightforward: it has a clearly defined structure, produces comparable results, and leaves many interesting questions open. You can incorporate it into your own experiments to investigate exactly what a modification enables sound to bring into the model and what additional capabilities it enables the LLM to use.
Text token embeddings were originally the gateway through which language models recognized text. ConvexLLM shows us that this gateway can also be opened to sound.
From this perspective, the boundary of text can also become a new starting point for exploring audio intelligence. article_adlist[
]article_adlistarticle_adlist[
Please contact this official account for permission before reposting.
Submissions or press inquiries: liyazhou@jiqizhixin.com ]article_adlist Statement: This is an exclusive Sina.com article; unauthorized reproduction is prohibited. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin
Non-targeted 300*250 button end
AI Trends