Towards Learning a Generalist Model for Embodied Navigation

Duo Zheng,Shijia Huang,Lin Zhao,Yiwu Zhong,Liwei Wang
2024-04-01
Abstract:Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve This paper aims to address the issue of **applying general models to embodied navigation tasks**. Specifically, researchers have found that most current embodied navigation models are designed for specific tasks and lack the ability to generalize to unseen scenarios. Although large language models (LLMs) have demonstrated exceptional capabilities in various tasks, their potential in the field of embodied navigation has not yet been fully explored. The paper proposes a new method—**NaviLLM**, which introduces pattern-based instructions to unify various tasks under a generative modeling framework, thereby leveraging pre-trained LLMs to handle different embodied navigation tasks. This approach not only integrates data sources from different datasets but also significantly improves the model's performance on multiple benchmarks and performs well on unseen tasks. In summary, the main goal of this paper is to build a general model that can be widely applied to various embodied navigation tasks and to validate its effectiveness in practical applications.