Abstract:We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation model.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to pre-train Vision-Language-Action (VLA) models using internet-scale video data without real robot action labels, so that these models can perform complex robotic tasks. Specifically, the paper proposes a method called "Latent Action Pretraining for general Action models" (LAPA), which aims to overcome the limitation of existing VLA models that require action labels collected through human teleoperation. This method not only leverages large-scale internet video data but also achieves good generalization across different environments and robot morphologies. ### Main Issues 1. **Utilization of Data Without Action Labels**: Existing VLA models typically require action labels collected through human teleoperation during the pre-training phase, which greatly limits the sources and scale of usable data. The LAPA method attempts to learn from internet videos without action labels to expand the amount of usable data. 2. **Generalization Across Environments and Robot Morphologies**: Existing models have limited generalization capabilities across different environments or robot morphologies. The LAPA method improves the model's generalization in this regard through latent action pretraining. 3. **Improving Pretraining Efficiency**: The LAPA method significantly improves pretraining efficiency by not requiring a large number of action labels during the pretraining phase. ### Solution 1. **Latent Action Quantization**: First, use the VQ-VAE (Vector Quantized Variational Autoencoder) objective function to learn discrete latent actions from continuous video frames. 2. **Latent Pretraining**: Then, use a pre-trained Vision-Language Model (VLM) to predict these latent actions based on video observations and task descriptions. 3. **Action Fine-Tuning**: Finally, fine-tune the model on a small scale of real robot operation data to map latent actions to actual robot actions. ### Experimental Results - **Performance Across Tasks, Environments, and Robot Morphologies**: Experimental results show that LAPA significantly outperforms baseline methods in these scenarios, especially in cross-environment and cross-robot morphology settings. - **Comparison with Existing Best Methods**: LAPA outperforms existing best VLA models (such as OPENVLA) on multiple tasks, particularly in tasks requiring language conditions and generalization to unseen objects and instructions. - **Pretraining Using Only Human Operation Videos**: Even when pretraining using only human operation videos, LAPA performs excellently, surpassing models pretrained with large-scale robot data. ### Conclusion The LAPA method provides an effective way to pre-train VLA models using large-scale internet video data, not only improving pretraining efficiency but also enhancing the model's generalization across different environments and robot morphologies. This opens up new possibilities for building general-purpose robotic foundation models.

Latent Action Pretraining from Videos

Masked Visual-Tactile Pre-training for Robot Manipulation

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

Knowledge-guided Pre-Training and Fine-Tuning: Video Representation Learning for Action Recognition

Learning to Act without Actions

Reinforcement Learning with Action-Free Pre-Training from Videos

TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation

Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training

Contrastive Language-Action Pre-training for Temporal Localization

OpenVLA: An Open-Source Vision-Language-Action Model

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization

Enhancing Video Transformers for Action Understanding with VLM-aided Training

Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust

AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition

Pre-trained Visual Dynamics Representations for Efficient Policy Learning

Probabilistic Vision-Language Representation for Weakly Supervised Temporal Action Localization

Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning

Video Language Planning