Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration,Abby O'Neill,Abdul Rehman,Abhinav Gupta,Abhiram Maddukuri,Abhishek Gupta,Abhishek Padalkar,Abraham Lee,Acorn Pooley,Agrim Gupta,Ajay Mandlekar,Ajinkya Jain,Albert Tung,Alex Bewley,Alex Herzog,Alex Irpan,Alexander Khazatsky,Anant Rai,Anchit Gupta,Andrew Wang,Andrey Kolobov,Anikait Singh,Animesh Garg,Aniruddha Kembhavi,Annie Xie,Anthony Brohan,Antonin Raffin,Archit Sharma,Arefeh Yavary,Arhan Jain,Ashwin Balakrishna,Ayzaan Wahid,Ben Burgess-Limerick,Beomjoon Kim,Bernhard Schölkopf,Blake Wulfe,Brian Ichter,Cewu Lu,Charles Xu,Charlotte Le,Chelsea Finn,Chen Wang,Chenfeng Xu,Cheng Chi,Chenguang Huang,Christine Chan,Christopher Agia,Chuer Pan,Chuyuan Fu,Coline Devin,Danfei Xu,Daniel Morton,Danny Driess,Daphne Chen,Deepak Pathak,Dhruv Shah,Dieter Büchler,Dinesh Jayaraman,Dmitry Kalashnikov,Dorsa Sadigh,Edward Johns,Ethan Foster,Fangchen Liu,Federico Ceola,Fei Xia,Feiyu Zhao,Felipe Vieira Frujeri,Freek Stulp,Gaoyue Zhou,Gaurav S. Sukhatme,Gautam Salhotra,Ge Yan,Gilbert Feng,Giulio Schiavi,Glen Berseth,Gregory Kahn,Guangwen Yang,Guanzhi Wang,Hao Su,Hao-Shu Fang,Haochen Shi,Henghui Bao,Heni Ben Amor,Henrik I Christensen,Hiroki Furuta,Homanga Bharadhwaj,Homer Walke,Hongjie Fang,Huy Ha,Igor Mordatch,Ilija Radosavovic,Isabel Leal,Jacky Liang,Jad Abou-Chakra,Jaehyung Kim,Jaimyn Drake,Jan Peters,Jan Schneider,Jasmine Hsu,Jay Vakil,et al. (192 additional authors not shown)

2024-06-01

Abstract:Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website <a class="link-external link-https" href="https://robotics-transformer-x.github.io" rel="external noopener nofollow">this https URL</a>.

Robotics

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the following issues:

1. **Cross-Robot Positive Transfer**: The researchers aim to verify whether the policies trained on data collected from various robots and environments can exhibit positive transfer effects, i.e., whether these policies perform better in evaluation environments compared to those trained solely on data from a single environment.

2. **Organization of Large-Scale Robot Learning Datasets**: To support future research on cross-robot models, the authors provide a large-scale robot learning dataset and standardize its processing so that other researchers can utilize these resources for further exploration.

Specifically, they constructed a large dataset by combining data from 21 research institutions, encompassing 22 different robot morphologies (totaling over 1 million trajectories), demonstrating 527 skills (approximately 160,266 tasks). Based on this dataset, they trained a high-capacity model called RT-X and demonstrated that this model improved its capabilities on various robot platforms by leveraging experiences from other platforms. Additionally, they provided two model architectures (RT-1-X and RT-2-X) to evaluate the performance of these policies and experimentally validated the model performance variations under different dataset scales.