Abstract:Most existing works in vision-and-language navigation (VLN) focus on either discrete or continuous environments, training agents that cannot generalize across the two. The fundamental difference between the two setups is that discrete navigation assumes prior knowledge of the connectivity graph of the environment, so that the agent can effectively transfer the problem of navigation with low-level controls to jumping from node to node with high-level actions by grounding to an image of a navigable direction. To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments. We refine the connectivity graph of Matterport3D to fit the continuous Habitat-Matterport3D, and train the waypoints predictor with the refined graphs to produce accessible waypoints at each time step. Moreover, we demonstrate that the predicted waypoints can be augmented during training to diversify the views and paths, and therefore enhance agent's generalization ability. Through extensive experiments we show that agents navigating in continuous environments with predicted waypoints perform significantly better than agents using low-level actions, which reduces the absolute discrete-to-continuous gap by 11.76% Success Weighted by Path Length (SPL) for the Cross-Modal Matching Agent and 18.24% SPL for the Recurrent VLN-BERT. Our agents, trained with a simple imitation learning objective, outperform previous methods by a large margin, achieving new state-of-the-art results on the testing environments of the R2R-CE and the RxR-CE datasets.

Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation

ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments

Waypoint Models for Instruction-guided Navigation in Continuous Environments

Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environments

Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation

Diagnosing Vision-and-Language Navigation: What Really Matters

Bird's-Eye-View Scene Graph for Vision-Language Navigation

Towards Navigation by Reasoning over Spatial Configurations

Narrowing the Gap between Vision and Action in Navigation

Active Perception for Visual-Language Navigation

Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas

Continual Vision-and-Language Navigation

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs

Navigation with VLM framework: Go to Any Language

AdaVLN: Towards Visual Language Navigation in Continuous Indoor Environments with Moving Humans

Scaling Data Generation in Vision-and-Language Navigation

Active Visual Information Gathering for Vision-Language Navigation

Learning to Navigate in Complex Environments