LeCun and Balestriero Publish JEPA Paper: 109-Million-Parameter Model Surpasses… in Cross-Modal Representation Alignment

At 66, Yann LeCun has returned to the front lines of AI research with yet another paper.

On October 29, LeCun and Randall Balestriero, an assistant professor of computer science at Brown University, jointly released a new 37-page paper, "Proving and Improving the Platonic Representation Hypothesis with JEPAs." The paper proposes that, under certain conditions, AI models of different modalities can learn mutually alignable world representations even when trained independently.

Even more striking is a set of numbers. A text JEPA trained by the research team with only 109 million parameters outperformed large language models including Qwen2.5-72B and Llama-3-70B in cross-modal representation alignment tests.

The difference in parameter scale is about 660 times.

In another experiment, a model using the new method needed only 21.6 million training samples to reach the representation alignment level that a next-token prediction model achieved with 900 million samples. The difference in sample count is about 42 times.

But this is not the conclusion the paper truly intends to convey.

LeCun's new paper challenges "scale is all you need": why can 109 million parameters challenge 72 billion?

In this paper, the authors LeCun and Balestriero seek to answer a question more fundamental than model size: when AI observes the world through different means such as text and images, what exactly determines whether they can form similar internal understandings?

For LeCun, who has long advocated the world model approach, this is also an important theoretical exploration.

The starting point of this research comes from a rather philosophical hypothesis proposed by an MIT research team two years ago.

In 2024, Minyoung Huh, Brian Cheung, Tongzhou Wang and Phillip Isola published "The Platonic Representation Hypothesis" at ICML.

The researchers observed an interesting phenomenon: as the capabilities of different models increase, the internal representation structures of image models and language models seem to be becoming increasingly similar.

For example, one can use a photo to describe a scene of a cat lying on a sofa, or use a sentence to express the same meaning. Images are made up of pixels, and text is made up of tokens; the two look completely different, but the real-world object they point to is the same. If AI truly learns some kind of shared structure behind these contents, then the representations of relationships among things inside different models may gradually converge.

The 2024 study observed that increasing model scale is often accompanied by cross-modal representation convergence, but it did not prove that this phenomenon necessarily occurs, nor did it explain what training methods can achieve it more efficiently.

LeCun and Balestriero attempt to take a step further. In the paper, they establish a mathematical framework that treats different modalities as observations of the same latent world state, and study under what conditions models can learn mutually aligned representations.

The answer is related to the positive sample pairs constructed during training.

The researchers call one type of data relationship Good Views, which can be understood as effective observations of the same scene.

For example, two photos of the same scene, or two passages of text separately describing the same scene, although their specific contents are not entirely identical, jointly point to a certain state in the real world. In contrast, the paper uses Bad Views to describe another type of training sample construction: for example, the prefix and subsequent content within the same passage of text, or different segments obtained by masking the same input.

The difference between the two lies in the fact that the former is more conducive to the model extracting information shared behind different observations; the latter may cause the model to spend more representation capacity on details unique to a single modality.

The paper proposes that when different JEPA models are trained through positive sample relationships that meet specific conditions, even if they process different modalities, they can learn representation spaces that differ only by a rotation transformation.

The so-called "rotation" does not mean changing the orientation of an image, but rather applying a mathematical transformation to the high-dimensional vectors inside the model so that the representations of different models can correspond to each other. This is a mathematical conclusion with clear preconditions, not a proof that all existing AI models already possess exactly the same understanding of the world.

The research team then tested this theory through experiments. The paper compares text models and vision models using different training strategies, focusing on how close their internal representation structures are. The results show that a text JEPA trained with Good Views performs outstandingly in cross-modal representation alignment.

The specific cross-modal representation alignment metric reported in the paper: the higher the score, the closer the model representation structures, and it is not a comprehensive capability score for language models.

Among them, the text JEPA with 109 million parameters scored 0.184, higher than Qwen2.5-72B's 0.154. This means that in this test, a small model with only about one six-hundredth the parameter count of a large language model instead learned a representational structure closer to that of vision models.

But note that the "surpassing" here has strict boundaries. It does not mean that JEPA's mathematical reasoning, code generation, or knowledge question-answering capabilities exceed those of Qwen2.5-72B. The paper compares cross-modal representation alignment, not the general large-model capabilities familiar to developers.

Another important experiment involved training efficiency.

Researchers found that under the same corpus and comparison settings, the Good Views training scheme required about 21.6 million samples to reach the representation alignment level achieved by next-token prediction with 900 million samples.

Calculated by sample count, the efficiency gain is about 42 times; but calculated by the actual number of text tokens read, the gap is about 14 times.

This suggests that what information relationships the model is taught to learn during training may be more important than simply having it process more text. Even more interestingly, the research team also attempted to turn this representation alignment capability into practical applications.

Through a linear transformation, they connected separately trained vision encoders and text models to build an image captioning system. On the COCO dataset, the system achieved a CIDEr score of 0.838. In another experiment that did not require paired image-text data to fit the alignment transformation, the model achieved 64% on the ImageNet-100 Top-5 classification metric, surpassing some of the language model approaches compared in the paper. This means the research results no longer remain at the level of mathematical proof, and also demonstrates the practical application space that cross-modal representation alignment may bring.

Leaving Meta to start a business at 66, why does LeCun still insist on JEPA?

Behind this paper is actually a continuation of LeCun's own research experience.

LeCun was born in France on July 8, 1960, and is now 66 years old. He once served as Meta's chief AI scientist and is also one of the pioneers of convolutional neural network research, sharing the 2018 ACM Turing Award with Geoffrey Hinton and Yoshua Bengio.

English text is being prepared.

As early as the 1980s, LeCun began exploring the application of neural networks in computer vision. He promoted the development of convolutional neural networks, enabling machines to more effectively recognize handwritten characters and image content. Later, these studies became an important foundation of modern computer vision and deep learning.

English text is being prepared.

He does not deny the practical value of LLMs, but repeatedly emphasizes that predicting the next token alone is not enough to build intelligent systems capable of understanding the physical world, remembering environmental states, reasoning, and planning complex actions. In his vision, future AI should not only learn language and knowledge through massive amounts of text, but should also, like humans and animals, learn the abstract structure of the world by observing the environment.

This is also the reason he has promoted JEPA (Joint-Embedding Predictive Architecture) for many years. Unlike training methods that require the model to reconstruct input details, JEPA focuses more on learning abstract representations and making predictions in representation space.

In November 2025, LeCun publicly announced that he was preparing to leave Meta and establish a new AI research company to continue advancing the Advanced Machine Intelligence (AMI) research plan. He clearly stated at the time that the goal of the new company was to drive the next round of AI breakthroughs: building systems capable of understanding the physical world, possessing continuous memory, and having reasoning and complex action planning capabilities.

In March 2026, AMI Labs completed about $1 billion in financing. Unlike many startups developing products around existing LLMs, AMI Labs takes world models as its core research direction.

This background gives this paper an additional layer of significance.

English text is being prepared.

There is a kind of scientific persistence worth noting here: in an industry that increasingly values model scale, capital investment, and the speed of product releases, LeCun is still trying to find the next breakthrough from the learning mechanism itself.

Although this paper has attracted attention because of Yann LeCun's name, another author, Randall Balestriero, is also worth introducing.

Balestriero is an assistant professor of computer science at Brown University, and his research directions include self-supervised learning, deep neural network theory, and representation learning. His research experience spans signal processing, deep network theory, and self-supervised learning.

He received his PhD from Rice University in 2021, and afterward worked with LeCun at Meta FAIR as a postdoctoral researcher.

In 2023, Balestriero joined Citadel, working on financial time series forecasting and representation learning-related work. He then returned to academia to conduct research at Brown University.

Industry reactions: some marvel, others question

After the paper was released, the most widely circulated claim was undoubtedly that "109 million parameters defeat 72 billion parameters."

On Hugging Face, the paper entered the daily trending papers list. Co-author Balestriero also personally appeared in the comments section to introduce the research results.

He particularly emphasized that the research team's 109-million-parameter text JEPA trained with Good Views achieved a score of 0.184 on the cross-modal representation alignment metric, surpassing Qwen2.5-72B's 0.154 and Llama-3-70B's 0.151.

When replying on X to comments related to the paper, Yann LeCun said that the reason today's large language models can answer a large number of questions is largely because they have accumulated massive amounts of declarative knowledge during training. He wrote:

"Large language models have accumulated a vast amount of declarative knowledge. That is why they can answer so many questions, and also why they need so many parameters, thereby occupying a large amount of memory."

LeCun's view also sparked discussion among netizens. Some people remain wait-and-see about this technical route. Netizen Sage Williams commented: "If your view is correct, then your model should ultimately win out. We will wait and see."

Others focused on the practical value this training method may bring. Peter Ebasone believes this is good news for small laboratories and individual researchers, and said the technology may have application potential in the field of genomics.

Another netizen, Gordon Lawton, raised a more specific technical question: if a model can learn the structure of the world through word co-occurrence in text and object co-occurrence in images, does the same also apply to visuals and sounds that appear simultaneously in video? After all, humans perceive the world through the combined action of vision and hearing.

These comments also reflect differing expectations from the outside world regarding the JEPA route: on the one hand, people hope to see this theoretical advantage ultimately translated into actual model capability; on the other hand, higher data efficiency and the possibility of learning from multimodal information such as vision and hearing have also led some developers to see new research directions.

Reference links:

This article comes from the WeChat public account "AI Frontline" (ID: ai-front), author: Dongmei, published by 36Kr with authorization.