Zeng Yan's New Paper Reveals ByteDance Seed's Concentrated Bet Strategy: Win One Modality First, Then Expand
36Kr reports that Zeng Yan, head of ByteDance Seed, publishing the Cola2 paper and her promotion reflect Zhang Yiming's "concentrated bet" strategy. This approach advocates establishing an advantage in a single modality such as video first, then expanding to other modalities, rather than exploring multiple lines in parallel. This aligns with ByteDance's long-term goals of merging departments and pursuing Omni multimodal capabilities and trillion-parameter large models.
Why read this
SelectedAI score 80
The article analyzes the Cola2 paper and internal personnel changes at ByteDance, revealing its long-term strategy of concentrating computing power on a single modality before expanding.
Topics
Body · AI translation
Zeng Yan, born after 1995, is rising rapidly.
Born in 1997, Zeng Yan joined ByteDance through campus recruitment in 2021 and later became the head of pre-training for Seedance. Letter AI provided a detailed report on her in the article "ByteDance's 'Luo Fuli', Holding Up Half the Sky for Seedance."
Rumors now suggest that Zeng Yan's title has been elevated to core head of Seedance. As early as March this year, Zeng Yan's rank was 3-2, but by November, it had reached 4-2, achieving two consecutive promotions in one year. Moreover, Zeng Yan has taken over an increasing amount of work from Zhou Chang, the former core head of Seed.
In ByteDance's Seed team, which boasts extremely high talent density, such speed is not common.
The reason may be that Zeng Yan's approach aligns more closely with Zhang Yiming's vision. As the project leader, the paper published by her team serves as the best evidence.
This paper is titled "Cola2: Using Diffusion Models and Continuous Latent Space Methods in Video Generation for Language Modeling," and its weights were released on Hugging Face in the first half of November.
The paper proposes a method that allows the model to process text by generating videos. According to the paper, this method not only saves computational power but also produces better generation results.
This paper also serves as a roadmap.
Starting from diffusion models, Zeng Yan believes in the route of scale effects, which is consistent with the concept of ultra-large parameter models personally led by Zhang Yiming, as leaked by foreign media, and also aligns with the long-termism proposed by Zhang Yiming at internal meetings within ByteDance.
In the current context of hunger for computing power, ByteDance can hardly support a strategy of parallel multi-line development. Therefore, ByteDance merged departments, shifting from multi-line horse racing to concentrated betting. The goal is to win first in one modality, and then let the winner take the methods to other modalities.
Currently mainstream large language models, such as DeepSeek, Qwen, and Doubao, all use the same method called "autoregression." Like playing idiom solitaire, based on what has already been written previously, they guess what the next token is, guess one after another, from left to right, popping out one by one.
This method is very successful, but it has a natural flaw: it cannot create an outline in advance like we do when writing articles.
When people write articles, they often think about what to say in each section first, and then organize the language. Autoregressive models think while writing, making do with whatever comes up, and must queue up; if the previous character hasn't been written, there's no hope of writing the next one.
Cola2's idea is to split the act of writing into two layers.
The first layer is to clarify "what to say" in a place called "continuous latent space"; the second layer is to translate the clarified meaning into specific characters one by one.
It is equivalent to the first layer being outlining, and the second layer writing the main text according to the outline.
The raw data of a video is very large; if used directly for training, it would lead to insufficient computing power. Therefore, the common practice for video generation models is to first compress the frames into a much smaller "archive," with the model working inside this archive, and finally decompressing it frame by frame.
The work done inside the archive is called "diffusion." This is similar to developing photos in the past: starting with a negative full of snowflakes (noise), the model gradually wipes away the noise step by step, and the image slowly emerges.
Cola2's logic is to treat "generating text" as "generating video."
This logic was already present in the first-generation Cola DLM, but Cola DLM had shortcomings. Cola2's paper states that Cola DLM is weak in generative tasks, and the problem lies with the decoder responsible for "writing the main text." Thus, Cola2's main role is to make up for this shortcoming; it does three additional things compared to the previous generation.
The first thing is learning to cut according to meaning.
To compress text like compressing video, one must first decide how to chunk it.
The old method was rigidly cutting by length, such as every two tokens per block. As a result, "Beijing Roast Duck" might be split into "Beijing" and "Roast Duck," or even into nonsensical chunks like "Jing Kao."
Cola2 changes this to dynamic cutting based on meaning, averaging two tokens compressed into one "outline point." Which specific tokens are grouped together is determined by the content.
How much to compress relies on a negative feedback mechanism similar to an air conditioner's thermostat. If compressed too tightly—for example, splitting "Bei," "Jing," "Kao," "Ya"—it automatically loosens slightly. If compressed too loosely—for example, "Beijing Roast Duck is delicious"—it tightens slightly, always stabilizing near the target.
In experiments, cutting by meaning achieved an accuracy rate 8.2% higher than cutting by fixed length. More interestingly, under appropriate settings, the performance after compression (29.4%) was slightly better than without any compression (28.7%). This indicates that what was removed was mainly "water" (redundancy).
The second thing is enabling the writer of the main text to both read from the outline and improvise.
If the decoder only reads from the outline, then once the outline makes a mistake, the entire article naturally goes wrong. Therefore, when training the decoder, Cola2 deliberately provides three types of outlines: one clear, one with added noise making it unclear, and one completely absent.
It is like training an actor: whether the teleprompter is clear, glitchy, or blacked out, they must still be able to continue acting.
The final selected ratio is that only 20% of cases provide a clear outline, while the remaining 80% are blurry or non-existent. Even so, the decoder's accuracy in restoring the original text from a clear outline remains above 99.5%.
Only when it comes time to actually write should you use the CFG knob to adjust the balance between "following the outline" and "improvising."
The third step is to train the outline and the main text together. This is what the subtitle Co-Adaptive refers to.
Previous approaches often involved training the "compressor" first, freezing it, and then training the model to write the main text.
If the outline template is fixed, even if problems with the outline are discovered during writing, the outline side remains completely unaffected. Cola2 simply lets both sides train together; if the part writing the main text feels something is insufficient, the signal is sent back, forcing the outline side to adjust.
In the experimental results, if the compressor is completely fixed and not allowed to be modified, the score is 47.6%. When both sides train together, it rises to 53.5%. Adding a training objective that allows the compressor to learn to "predict the following text" raises the score to 58.2%.
There is also an counter-intuitive design here. The DiT responsible for generating outlines is only allowed to learn, not to modify the compressor in return.
Cola2 used approximately 346 billion tokens for training. The model has not yet converged, achieving an average score of 60.10 across 8 tasks, surpassing Llama-3-8B on HellaSwag and StoryCloze. Meanwhile, Llama-3's public pre-training volume exceeds 15 trillion tokens, which is more than 43 times that of Cola2.
Regarding computing power, according to the paper, because on average two tokens are compressed into one outline point, the sequence length DiT needs to process is halved. Theoretically, its main computational load drops to half, and the attention component drops to a quarter.
The final section of the paper states that the model is "far from convergence," especially the DiT. The training scale is "limited by available resources." To reach the upper limit of this path, larger models, longer training, more data, and more computing power are required.
Additionally, the paper suggests that continuous latent spaces may become a shared interface between text, images, and video, leading to "the unification of understanding and generation."
"The unification of understanding and generation" is precisely the story Zhou Chang has been trying to explain at ByteDance for over a year but failed to make clear.
Zeng Yan's approach aligns better with Zhang Yiming.
On November 12, The Paper reported that Zhang Yiming recently said at an all-hands meeting for the AI team that ByteDance has the ability to compete with OpenAI and Anthropic, but truly catching up to the frontier would take "at least until 2029 or 2030."
Zhang Yiming stated that compared to top overseas companies, the pre-training data scale of the Seed model "is still lacking," which is the most pressing issue to address at present. At the same time, foreign media reported that Seed is pre-training a model with parameters reaching up to 10 trillion.
Insufficient data, the need to scale models up, and a commitment to long-termism. This clearly aligns with Zeng Yan's approach. Zhou Chang's approach is quite different.
In July 2025, Yang Jianzhao, who was previously responsible for visual generation, left the company. The lines for image generation and video generation were handed over to Zhou Chang. Since then, he has been promoting a new narrative within ByteDance called "Unified Multimodal Understanding and Generation."
Zhou Chang's idea was to start from understanding and incorporate generation as well, enabling a single model to both see and create.
But this story didn't pan out.
The most prominent result reflecting this unified approach after Zhou Chang took over was the image model Seedream 5.0 Lite, released in February alongside Seedance 2.0.
Internally at ByteDance, the evaluation of 5.0 Lite was "a relatively small model," with "room for improvement" in structural stability, photorealism, and aesthetics. The general consensus in the community is that Zhou Chang understands understanding well but is less familiar with generation.
A researcher from Alibaba's Qwen team stated that there are currently no known highly effective solutions for "unifying understanding and generation," and no one can achieve it yet. In such situations, simply stacking parameters may not yield results.
In August this year, Seed underwent an organizational adjustment. Zhou Chang's foundational VLM (Vision-Language Model) was merged into the large language model system, changing from an independent department to a sub-department focused on data. An insider at Seed said: "He lost his stronghold; essentially, he no longer has core territory."
Actually, Zeng Yan also came from a background in understanding. Just two months after joining in 2021, she published X-VLM as the first author, researching how to make AI correlate objects in images with text.
For example, if you say "a little girl in red clothes is riding a bicycle," the model must accurately locate that little girl in the image.
After transferring to Seed in 2023, she began moving towards generation. In 2024, Zeng Yan participated in PixelDance, which solved the problem of characters deforming when they move in videos. It is now the core technical foundation for ByteDance's video generation and the origin of the name Seedance.
The success of Seedance 2.0 is not directly related to "unified understanding and generation." Instead, it generates video and audio together within a single DiT (Diffusion Transformer), ensuring that visuals and sound naturally align. The capabilities of Seedance 2.0 mainly stem from Zeng Yan's expertise in "continuity."
Therefore, some insiders at Seed indicate that Zeng Yan is the actual leader of Seedance, with technology and decision-making resting at her level.
The training objectives for understanding and generation are different, the data looks different, and the calculation methods are also different. If you force them into a single model, the two sides tend to hold each other back.
It’s like Zhou Botong’s Left-Hand Right-Hand Martial Arts in "The Legend of the Condor Heroes": drawing a circle with the left hand is simple, drawing a square with the right hand is simple, but doing both together is extremely difficult.
At the end of 2025, the Seedance team once had a dinner gathering to discuss the training plan for their next-generation model. Zeng Yan proposed training a larger model with at least 200 billion parameters (200B). Some team members felt this was too aggressive and suggested that training a 100B model first would be more prudent, but Zeng Yan insisted on achieving it in one step. Later, at Seed's formal review meeting, both Wu Yonghui and Zhou Chang supported her proposal.
Previously, there were leaks from overseas sources claiming that Zhang Yiming was leading ByteDance in building a trillion-parameter large model and that he, as China's richest man, was personally involved in the project. Clearly, Zeng Yan's approach aligns more closely with what Zhang Yiming wants to achieve.
Finally, there is another layer: background.
Zhou Chang was poached by ByteDance with an eight-figure annual salary. When he joined, he brought over a dozen former colleagues from Alibaba, which even led to non-compete arbitration disputes with Alibaba. He has been at ByteDance for two years, and rumors of his departure have surfaced at least twice.
English text is being prepared.
Zeng Yan, on the other hand, is a campus recruit hired directly by ByteDance. She reached level 4-2 within five years, staying consistently on the Seedance track throughout. For ByteDance, there is a clear distinction in closeness between Zhou Chang and Zeng Yan.
ByteDance is shifting from multi-track horse racing to concentrated betting.
Whenever ByteDance enters a new field, it usually establishes more than one team. Zhang Yiming prefers to deploy several teams simultaneously for parallel exploration, backing whoever succeeds. Douyin was developed this way.
The same applies to large models.
Over the past two years, Seed recruited the largest group of top-tier researchers domestically, distributed across various directions such as text, code, vision, speech, and reasoning. According to reports from LatePost, some were promised that they could lead the "tide" through their own innovation here.
A researcher who previously worked at Seed said, "Everyone wants to accomplish something significant."
Computing power is the biggest problem.
Large-parameter models consume the most computing power, which is also the scarcest and most precious resource.
If a family has many children and was previously well-off, each child could attend several extracurricular classes. Now, if you want one child to attend a top prestigious school and achieve outstanding results to stand out, then other average-performing children will have to reduce their extracurricular activities.
The organizational adjustment organized by ByteDance in August mentioned earlier follows this logic.
Seed's foundational model team newly established four first-level departments: Pre-training Data, Reinforcement Learning, Post-training for Office Applications, and Post-training for Conversations. The heads of these departments report directly to Wu Yonghui.
The announcement stated: 'Merge similar work, reduce overlap, and consolidate internally.'
Previously, Seed organized teams by modality, with text, code, vision, and reasoning each having their own data teams and post-training teams.
This approach facilitated rapid capability building in the early stages, but as the number of directions increased, business overlaps between teams emerged. Some directions saw duplicated investments, and cross-team communication costs rose significantly.
Scaling up models is not simply about adding parameters; it is a systematic engineering challenge.
As parameters increase, more and better data must keep pace; otherwise, the additional parameters merely relearn what has already been learned. This leads to diminishing returns, making it difficult to achieve meaningful gains even with substantial financial investment.
At parameter scales of 5 trillion or 10 trillion, there is no room for each modality to maintain separate data teams and conduct independent post-training processes.
Furthermore, ByteDance aims for its next-generation models to directly adopt an Omni (full-modality) architecture, integrating language, code, vision, and speech into a single foundational base.
Consequently, decisions regarding the volume of data, the timing of its introduction, and how to maintain balance will all impact final capabilities.
An industry practitioner told Wan Dian: "Omni is the trend of the industry."
In July this year, Tencent merged its multimodal model department into Yao Shengyu's large language model department. Starting from Qwen3.5, Alibaba integrated vision capabilities directly into its mainline models, ceasing to release separate general-purpose vision understanding models. Similarly, DeepSeek V4.1 Flash natively integrates vision understanding.
Seedance has demonstrated that leading model capabilities can quickly translate into user growth and API revenue.
Of course, ByteDance is not solely focused on immediate revenue.
Zhang Yiming stated at an internal Seed meeting that the company could accept the model lagging behind for a period, hoping the team would prioritize pursuing the upper limits of intelligence.
He also emphasized that developing models requires long-termism and delayed gratification, rather than trading others' outputs for temporary leaderboard rankings.
Reports from early November indicated that overseas revenue accounted for 50% of Seedance's total, with overseas revenue nearly quintupling over the past six months.
However, on the most critical battlefield of foundational models, Seed remains in a catch-up position. Particularly in programming and Agent capabilities, it lags significantly behind startups such as Moonshot AI and Zhipu AI.
Zhang Yiming believes that it will take "at least until 2029 or 2030" for ByteDance to catch up with frontier models.
Therefore, ByteDance's selection of Zeng Yan essentially represents a bet on a specific path: first win in one modality, then have the victors expand their winning methods to other modalities.
This was also how ByteDance built its foundation. They first validated recommendation algorithms on Toutiao, then applied this strategy to Douyin.
This article comes from the WeChat public account "Letter AI", author: Miao Zheng, published with authorization from 36Kr.
AI Trends