How do we teach AI to truly understand video? V-JEPA offers a new answer: by predicting features, not just pixels. We'll break down this fascinating technique, explaining how it helps AI learn more robust and meaningful visual representations from video. Join us to explore how V-JEPA is pushing the boundaries of video AI.

This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model’s parameters; e.g., using a frozen backbone, our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.

References:

This episode draws primarily from the following paper:

Revisiting Feature Prediction for Learning VisualRepresentations from Video

Adrien Bardes, Quentin Garrido, Jean Ponce, XinleiChen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas

The paper references several other important works in this field. Please refer to the full paper for acomprehensive list.

Disclaimer:

Please note that parts or all this episode was generatedby AI. While the content is intended to be accurate and informative, it isrecommended that you consult the original research papers for a comprehensiveunderstanding.