Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics

Minttu Alakuijala,Reginald McLean,Isaac Woungang,Nariman Farsad,Samuel Kaski,Pekka Marttinen,Kai Yuan

2024-05-30

Abstract:Natural language is often the easiest and most convenient modality for humans to specify tasks for robots. However, learning to ground language to behavior typically requires impractical amounts of diverse, language-annotated demonstrations collected on each target robot. In this work, we aim to separate the problem of what to accomplish from how to accomplish it, as the former can benefit from substantial amounts of external observation-only data, and only the latter depends on a specific robot embodiment. To this end, we propose Video-Language Critic, a reward model that can be trained on readily available cross-embodiment data using contrastive learning and a temporal ranking objective, and use it to score behavior traces from a separate reinforcement learning actor. When trained on Open X-Embodiment data, our reward model enables 2x more sample-efficient policy training on Meta-World tasks than a sparse reward only, despite a significant domain gap. Using in-domain data but in a challenging task generalization setting on Meta-World, we further demonstrate more sample-efficient training than is possible with prior language-conditioned reward models that are either trained with binary classification, use static images, or do not leverage the temporal information present in video data.

Robotics,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

This paper aims to address the problem of how to enable robots to perform tasks through natural language instructions. Current methods often require a large amount of language annotation and demonstration specific to the robot. The paper proposes a reward model called Video-Language Critic (VLC), which can utilize cross-modal data for pre-training and evaluate behavior trajectories without actual operational data. VLC is trained through contrastive learning and temporal ordering objectives to score behavior trajectories. Experiments show that VLC is more efficient in policy training for Meta-World tasks compared to using sparse rewards alone, even in the presence of significant domain gaps. Furthermore, VLC achieves more effective training on unseen tasks compared to previous language-conditioned reward models that rely on binary classification, static images, or do not utilize temporal information in videos. The goal of the paper is to create a universal robotic system that can self-evaluate behavior and learn reward functions, in order to improve the quality and diversity of skills.