joint-embedding predictive architecture
In plain wordsA way of training AI on images or video in which the model predicts an abstract summary of a hidden part, rather than every pixel, so it learns what matters and ignores unpredictable detail.
A self-supervisedself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image. architecture in which a context encoder embeds the visible part of an input, a target encoder embeds a hidden part (a masked image region or future video frames), and a predictor network is trained to produce the target's embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. from the context's, with the loss measured in representation space rather than in pixels. Predicting abstract representations lets the model ignore inherently unpredictable details, such as the exact texture of foliage, which generative reconstruction objectives must model. Collapse to trivial constant embeddings is avoided by updating the target encoder as an exponential moving average of the context encoder, or by regularizing the statistics of the embeddings. Yann LeCun proposed JEPAs as the building blocks of predictive world modelsworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real. for autonomous machine intelligence. Instances include I-JEPA (images, 2023), V-JEPA (video, 2024) and V-JEPA 2 (2025), which Meta reported could be adapted with a small amount of robot data to plan simple manipulation tasks.
Unlike the masked autoencoder, the JEPA never reconstructs pixels; it predicts the embedding of the masked block.
Origin
Proposed by Yann LeCun in the position paper 'A Path Towards Autonomous Machine Intelligence' (OpenReview, June 2022). The first large-scale implementation was I-JEPA, by Mahmoud Assran and colleagues at Meta (arXiv January 2023; CVPR 2023).
Usage note
LeCun presents JEPAs as an alternative to generative, autoregressive models on the route to more capable AI; that broader claim is debated, but the architecture itself is an established self-supervised method.
Related
- self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
- DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
- generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
Sources
- LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. OpenReview preprint, version 0.9.2.
- Assran, M. et al. (2023). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR 2023.
Mentioned in
DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself., energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.