Balestriero and LeCun turn the Platonic Representation Hypothesis into a theorem

Researchers Randall Balestriero and Yann LeCun posted a paper arguing that the long-standing idea that AI models trained on different types of data eventually converge on the same internal representation of the world, known as the Platonic Representation Hypothesis, can now be proven mathematically rather than just observed. They present what they call a Platonic Representation Theorem, which applies to Joint-Embedding Predictive Architectures, or JEPAs: when a JEPA is trained separately on different modalities using what the authors call good views, meaning two observations of the same underlying scene, the resulting representations end up aligned up to a single rotation.

The paper also argues that next-token prediction, the standard way large language models are trained, is itself a kind of JEPA, but one that uses bad views, meaning parts of a single document rather than two observations of a shared scene. The authors say this explains why language models only align with vision models as they scale up, rather than aligning efficiently from the start.

Testing the theory on data ranging from 145,000 captions to 940 million web images, the authors report that good views need 42 times fewer training samples than next-token prediction to reach the same level of alignment. They also report that a small, 109-million-parameter text-only JEPA aligned better with the DINOv2 and CLIP image encoders than any language model they tested, up to 72 billion parameters.

Why it matters

If correct, the theory suggests a more sample-efficient path to building AI systems that unify language and vision than simply scaling up next-token prediction.

Source: arXiv