Abstract:LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) task. However, existing LLM-based methods often focus only on solving high-level task planning by selecting nodes in predefined navigation graphs for movements, overlooking low-level control in navigation scenarios. To bridge this gap, we propose AO-Planner, a novel Affordances-Oriented Planner for continuous VLN task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making, both performed in a zero-shot setting. Specifically, we employ a Visual Affordances Prompting (VAP) approach, where the visible ground is segmented by SAM to provide navigational affordances, based on which the LLM selects potential candidate waypoints and plans low-level paths towards selected waypoints. We further propose a high-level PathAgent which marks planned paths into the image input and reasons the most probable path by comprehending all environmental information. Finally, we convert the selected path into 3D coordinates using camera intrinsic parameters and depth information, avoiding challenging 3D predictions for LLMs. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance (8.8% improvement on SPL). Our method can also serve as a data annotator to obtain pseudo-labels, distilling its waypoint prediction ability into a learning-based predictor. This new predictor does not require any waypoint data from the simulator and achieves 47% SR competing with supervised methods. We establish an effective connection between LLM and 3D world, presenting novel prospects for employing foundation models in low-level motion control.

Multimodal LLM Guided Exploration and Active Mapping using Fisher Information

Line feature guided exploration for active mapping

FIT-SLAM -- Fisher Information and Traversability estimation-based Active SLAM for exploration in 3D environments

FLAME: Learning to Navigate with Multimodal LLM in Urban Environments

Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Intelligent LiDAR Navigation: Leveraging External Information and Semantic Maps with LLM as Copilot

Navigate Complex Physical Worlds via Geometrically Constrained LLM

MM-Gaussian: 3D Gaussian-based Multi-modal Fusion for Localization and Reconstruction in Unbounded Scenes

MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements

L3MVN: Leveraging Large Language Models for Visual Target Navigation

Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

Active Visual Information Gathering for Vision-Language Navigation

A New Multimodal Map Building Method Using Multiple Object Tracking and Gaussian Process Regression

Active Semantic Mapping and Pose Graph Spectral Analysis for Robot Exploration

Embodied AI in Mobile Robots: Coverage Path Planning with Large Language Models

Vision and Language Navigation in the Real World via Online Visual Language Mapping

VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation

Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models