An Embodied Generalist Agent in 3D World

Jiangyong Huang,Silong Yong,Xiaojian Ma,Xiongkun Linghu,Puhao Li,Yan Wang,Qing Li,Song-Chun Zhu,Baoxiong Jia,Siyuan Huang

2024-05-10

Abstract:Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the limitations of current general artificial intelligence models in handling tasks in the 3D world. Specifically, most existing models primarily rely on 2D image inputs and have limited capabilities in understanding and handling 3D environment tasks such as 3D localization, embodied reasoning, and action. These limitations hinder the models from performing real-world tasks and achieving general intelligence.

To address these issues, the authors propose an embodied multimodal general agent named LEO. LEO is capable of handling first-person perspective 2D images, 3D point clouds, and text inputs, and performs tasks such as perception, localization, reasoning, planning, and action under a unified task interface, model architecture, and objectives. Additionally, the paper details a large-scale dataset used for training LEO and a pipeline that leverages large language models to assist in generating high-quality 3D vision-language data. Through extensive experimental validation, LEO demonstrates its superior performance in various 3D tasks, including 3D captioning, question answering, embodied reasoning, navigation, and manipulation. The research also provides valuable insights for future development of embodied general agents through ablation analysis.