Abstract:Vision-language navigation (VLN) is a critical domain within embedded intelligence, requiring agents to navigate 3D environments based on natural language instructions. Traditional VLN research has focused on improving environmental understanding and decision accuracy. However, these approaches often exhibit a significant performance gap when agents are deployed in novel environments, mainly due to the limited diversity of training data. Expanding datasets to cover a broader range of environments is impractical and costly. We propose the Vision-Language Navigation with Continual Learning (VLNCL) paradigm to address this challenge. In this paradigm, agents incrementally learn new environments while retaining previously acquired knowledge. VLNCL enables agents to maintain an environmental memory and extract relevant knowledge, allowing rapid adaptation to new environments while preserving existing information. We introduce a novel dual-loop scenario replay method (Dual-SR) inspired by brain memory replay mechanisms integrated with VLN agents. This method facilitates consolidating past experiences and enhances generalization across new tasks. By utilizing a multi-scenario memory buffer, the agent efficiently organizes and replays task memories, thereby bolstering its ability to adapt quickly to new environments and mitigating catastrophic forgetting. Our work pioneers continual learning in VLN agents, introducing a novel experimental setup and evaluation metrics. We demonstrate the effectiveness of our approach through extensive evaluations and establish a benchmark for the VLNCL paradigm. Comparative experiments with existing continual learning and VLN methods show significant improvements, achieving state-of-the-art performance in continual learning ability and highlighting the potential of our approach in enabling rapid adaptation while preserving prior knowledge.

REVE-CE: Remote Embodied Visual Referring Expression in Continuous Environment

Object-aware Navigation for Remote Embodied Visual Referring Expression.

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments.

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

March in Chat: Interactive Prompting for Remote Embodied Referring Expression

Scene-Intuitive Agent for Remote Embodied Visual Grounding

Visionary: vision-aware enhancement with reminding scenes generated by captions via multimodal transformer for embodied referring expression

Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression

Room-Object Entity Prompting and Reasoning for Embodied Referring Expression

Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environments

Reinforced Structured State-Evolution for Vision-Language Navigation

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

Embodied Referring Expression for Manipulation Question Answering in Interactive Environment

MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains

Vision-Language Navigation with Continual Learning

Towards Further Comprehension on Referring Expression with Rationale

Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigation

Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration

Continual Vision-and-Language Navigation

RREx-BoT: Remote Referring Expressions with a Bag of Tricks