LeCun team's paper uses JEPA to prove the Platonic representation hypothesis and improve sample efficiency

AI pioneer Yann LeCun has long argued that "generative AI is a dead end, and world models are the real future."

He believes that autoregressive generation has a fundamental mathematical flaw, that text cannot carry most of human knowledge, that true reasoning is not optimization-based inference, and that the measure of intelligence should be the speed of learning new tasks and the ability to handle entirely new environments with zero shots.

Based on this view, LeCun seeks an alternative route: world models + JEPA, a goal-driven architecture. A paper he recently submitted takes a big step in this direction: it mathematically argues that "the next-token prediction route is inefficient" and also presents his own method, using a 109-million-parameter model to beat a 72-billion-parameter LLM.

Would a model that only looks at images and a large text model form similar "concept maps" internally?

The Platonic Representation Hypothesis (PRH) was proposed by Huh et al. in an ICML 2024 paper. It holds that models trained separately on different modalities (vision, language) will, as their scale increases, have increasingly similar internal representations, eventually converging to the same shared statistical model of reality. This is like different people in Plato's cave seeing the same "world of ideas." But it had only empirical curves and an intuitive argument from an idealized contrastive learner—neither a proof nor a recipe—so the industry could only keep scaling up.

What this paper does is turn it into a theorem and give operable sufficient conditions.

Paper: "Proving and Improving the Platonic Representation Hypothesis with JEPAs"

Letting models learn the same scene from different descriptions

Every sentence a text model reads mixes two kinds of information: what it describes, and how it describes it. A dog chasing a ball belongs to the former; whether one uses "chasing" or "pursuing" belongs to the latter. If we want text models and image models to form compatible representations, we need text models to retain more of the content of the scene.

But during training, how do we tell the model which content is worth retaining?

The authors' approach is to have the model learn different observations of the same scene at the same time. Two people describing the same photo may vary in wording and sentence structure, but the subjects and relations in the scene are largely the same. If the model is required to extract similar representations from these descriptions, it gets a learning signal: find the common content behind different expressions.

JEPA (Joint Embedding Predictive Architecture) provides a way to implement this idea. In this paper's setup, an encoder first converts inputs into vectors, and training then pulls the vectors of a group of inputs closer together. This means that which inputs are chosen to form a group directly affects what the model is encouraged to retain.

To make the model extract scene information from text, should training show it different descriptions of the same scene, or different versions of the same sentence obtained by truncation and masking? The authors compared these two choices through controlled experiments.

Suppose a photo shows a dog chasing a ball on the grass. Two people write descriptions respectively: "A dog is chasing a ball on the grass" and "The puppy on the lawn is pursuing the toy."

In the first group, fragments are taken from these two different descriptions and some words are masked. For example, the model might see "the dog is chasing [MASK] on the grass" and "the puppy on the lawn is pursuing [MASK]." The two sentences express things differently; to make their vectors close, the model needs to make more use of the scene content shared behind "grass" and "lawn," "chasing" and "pursuing."

The second group takes only the first sentence and processes it into two versions, such as "A dog is [MASK] chasing a ball" and "A dog is chasing [MASK] on the grass." Both inputs retain identical wording such as "A dog is" and "chasing," and the model can use these textual similarities to bring their vectors close.

The authors call the first group "good views" and the second group "bad views." Here a "view" can be understood as one observation provided to the model during training. Both groups use the same JEPA learning method, allowing the authors to examine what difference it makes to the representation whether the inputs share a scene or the expression of a particular sentence.

"Good views" turn PRH into a theorem and improve the sample efficiency of the alignment process by 42 times.

This training scheme also needs to solve a problem: if the model converts all text into the same vector, it can also satisfy the requirement that "vectors be close." Therefore, the authors add a constraint to prevent "representation collapse," keeping the overall representation dispersed. The model must both find the common content of a group of descriptions and avoid conflating all scenes.

Along this line of thought, the connection between text and images also becomes easier to understand. Wording is information specific to text, shooting angle is information specific to images, and the subjects and relations in a scene may be jointly retained by both kinds of observation. If separately trained models both learn to extract these common structures, their internal representations have a chance to be compatible.

A caption generated by a network that has never been trained on any image-text pairs: it uses frozen DINOv2 and DINOv3 image encoders, is rotated only once, and employs a language model fine-tuned on text only; a new caption is generated at every scene change.

The authors therefore give mathematical conditions for this. In the finite world of the theoretical construction, they assume that each kind of observation is sufficient to determine the corresponding world state. If the training of different modalities connects the same relations among world states, then after reduction to world states, the models are solving the same optimization problem. Under the condition that the optimum is unique up to a rotation, the resulting representations then differ only by a single global rotation.

It is like two people each drawing a map, with different orientations, but with consistent relative positions among the cities, so that after rotation the maps can coincide.

This also leads to a comparison with next-token prediction: what kind of learning signal does predicting the following word give the model?

To continue the text accurately, the model needs to use the content, and also the grammar, wording, and style. By rewriting the objective function, the authors bring next-token prediction into a JEPA-like framework, and point out that what it connects is the prefix and the subsequent content of the same document. This connection preserves the document's own expressive characteristics.

On this basis, the authors propose an explanation: good-view training directly connects different descriptions of the same scene; next-token prediction, in contrast, needs to gradually establish connections among these descriptions from a large amount of linguistic associations. For forming representations that can be aligned with visual models, the latter may require more data.

The next thing the experiments need to test is how large a difference in effect this difference in training signal actually brings.

How much change can a different training method bring?

If the preceding explanation holds, then, with the same model size and training corpus, changing the choice of views should change how close the model is to visual representations.

The team first tested this on Flickr30k. Each image in this dataset has five human-written descriptions, which can provide different expressions of the same scene and can also be used to process multiple versions from a single-sentence description, making it suitable for comparing the difference between good views and bad views.

The researchers used the same text encoder and description corpus to train a good-view JEPA, a bad-view JEPA, and a next-token prediction model respectively, and then compared them with an independently trained visual encoder.

How can one judge whether they have learned similar structures? One way is to compare the "neighbors" each selects. For the same batch of image-text samples, the visual model finds similar images for an image, and the text model finds similar descriptions for a description. If the neighbors selected on the two sides often correspond, it shows that they organize these scenes in a relatively consistent way. The mutual 10-NN metric used in the paper measures exactly this degree of neighborhood overlap.

The results show that the text representations obtained by good-view JEPA are more consistent with visual representations.

In the Flickr30k experiment, good-view JEPA led on both neighborhood overlap and image-text retrieval metrics after rotation mapping.

However, good-view training also uses an extra piece of information: which descriptions belong to the same image. Could the improvement mainly come from this layer of association?

To this end, the authors concatenated two descriptions of the same image and gave them to the next-token prediction model to learn, and the improvement was still small. This shows that after providing related descriptions, one still needs to consider how to use them for training: putting two descriptions together for continuation and directly requiring them to form similar representations do not bring the same effect.

The next question is whether this advantage can extend to larger-scale web text from more complex sources.

The team used Recap-DataComp data, in which each image corresponds to the original alt text and two descriptions generated by LLaVA. These texts provide multiple views of the same image, and the text model reads the text during training.

Here, the authors further compared the number of training samples required to reach a similar level of alignment. Using the same 109-million-parameter encoder and the same corpus source, good-view JEPA achieved a WIT alignment score of 0.109 after processing 21.6 million training samples; next-token prediction achieved 0.111 after processing 900 million samples.

The scores of the two evaluated results are close, while the sample sizes differ by about 42 times. Since good-view training reads three texts per sample, calculated by the text tokens actually read, the gap is about 14 times. This grounds the impact of the training method in data utilization efficiency.

So, if training continues to be scaled up, what level can good-view JEPA reach?

Using text corresponding to 940 million web images, the team scaled training of the 109-million-parameter model to 900 million samples. On the WIT-1024 benchmark, with DINOv2-B as the visual reference, the default configuration achieved an average alignment score of 0.184 over three training runs, higher than Qwen2.5-72B's 0.154 and Llama-3-70B's 0.151.

This is the concrete result behind "109 million parameters surpassing 72 billion": the text neighborhood structure learned by the small model is more consistent with this independently trained visual model.

A comparison at the same scale once again demonstrates the role of the training method. Using the same encoder, the same text, and the same number of training samples, next-token prediction obtained 0.111 and bad-view JEPA 0.088, both lower than good-view JEPA's 0.184. When the visual reference was changed to CLIP, or the COCO test set was used instead, good-view JEPA still maintained its lead.

The performance of good-view training in the web-text scaling experiment.

At this point, the experiments support the preceding explanation from three levels: changing the training method can improve alignment, can reduce the samples needed to reach a similar level, and can also enable a smaller model to obtain a representation structure closer to that of a visual model than that of large language models.

Can this structural closeness further become functionality?

The authors demonstrated this with an image captioning task. First, a text decoder is trained to reconstruct descriptions from the vectors of the text JEPA. When processing an image, a frozen visual encoder extracts a vector, which is transformed into the text representation space through a linear mapping and handed to the decoder to generate a sentence.

Among these, the mapping is fitted using image-text paired data, and the decoder is trained on text. After further combining components such as DINOv2 and DINOv3, the system achieved a CIDEr score of 0.838 on COCO, a metric that measures the degree of consistency between generated descriptions and human-written descriptions.

The separately learned visual and text modules are thus connected: the scene information provided by the image enters the text representation space through the mapping and is finally read out as a description.

The COCO image captioning case shown by the authors. The separately learned visual and text modules are connected through a mapping, converting the scene information in the image into a textual description.

Conclusion

English text is being prepared.

Yann LeCun's research provides a direction worth further verification for multimodal models.

English text is being prepared.