Vision-Language Navigation with Embodied Intelligence: A Survey

Peng Gao,Peng Wang,Feng Gao,Fei Wang,Ruyue Yuan

2024-02-22

Abstract:As a long-term vision in the field of artificial intelligence, the core goal of embodied intelligence is to improve the perception, understanding, and interaction capabilities of agents and the environment. Vision-language navigation (VLN), as a critical research path to achieve embodied intelligence, focuses on exploring how agents use natural language to communicate effectively with humans, receive and understand instructions, and ultimately rely on visual information to achieve accurate navigation. VLN integrates artificial intelligence, natural language processing, computer vision, and robotics. This field faces technical challenges but shows potential for application such as human-computer interaction. However, due to the complex process involved from language understanding to action execution, VLN faces the problem of aligning visual information and language instructions, improving generalization ability, and many other challenges. This survey systematically reviews the research progress of VLN and details the research direction of VLN with embodied intelligence. After a detailed summary of its system architecture and research based on methods and commonly used benchmark datasets, we comprehensively analyze the problems and challenges faced by current research and explore the future development direction of this field, aiming to provide a practical reference for researchers.

Artificial Intelligence,Computer Vision and Pattern Recognition,Robotics

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper primarily explores the research progress and future directions of Vision-Language Navigation (VLN) in the field of Embodied Intelligence. Specifically:

1. **Core Objectives**:
- Enhance the perception, understanding, and interaction capabilities of agents in environments, enabling them to effectively communicate with humans through natural language and understand and execute navigation tasks based on language instructions.

2. **Technical Challenges**:
- Multimodal Information Fusion: How to effectively combine visual information with language instructions.
- Generalization Ability: Improve the adaptability and generalization ability of agents in different environments.
- Data Limitations: The limitations of existing datasets and the gap between simulated environments and real-world environments.

3. **Research Progress**:
- Since 2012, with the development of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), VLN has gradually shifted from theoretical exploration to practical application.
- In 2017, Chang et al. introduced the Matterport 3D simulated scene dataset, providing a benchmark testing environment for VLN.
- In 2018, Anderson et al. first proposed the concept of VLN and created the first VLN benchmark dataset—Room-to-Room (R2R).
- Subsequently, many innovative methods have emerged, such as Transformer-based VLN agents and self-supervised imitation learning, significantly improving the performance of agents.

4. **Future Prospects**:
- Provide a comprehensive research review, analyze the existing problems and challenges in current research, and explore future development directions.
- Aim to provide practical references for researchers and promote the development of the VLN field.

Through this review, the authors hope to provide readers with a clear understanding of the current status and development trends in the VLN field and guide future research work.