Abstract:Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest. Most research has focused on ground-based agents, while UAV-based VLN remains relatively underexplored. Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navigation tasks between ground and aerial environments. To address these disparities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology. To enable realistic UAV trajectory simulation in VLN tasks, we propose the OpenUAV platform, which features diverse environments, realistic flight control, and extensive algorithmic support. We further construct a target-oriented VLN dataset consisting of approximately 12k trajectories on this platform, serving as the first dataset specifically designed for realistic UAV VLN tasks. To tackle the challenges posed by complex aerial environments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help, which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks. We also propose a UAV navigation LLM that, given multi-view images, task descriptions, and assistant instructions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajectory generation. The evaluation results of our method significantly outperform the baseline models, while there remains a considerable gap between our results and those achieved by human operators, underscoring the challenge presented by the UAV-Need-Help task.

Demo Abstract: Embodied Aerial Agent for City-level Visual Language Navigation Using Large Language Model

NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation

AerialVLN: Vision-and-Language Navigation for UAVs

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

Aerial Vision-and-Language Navigation via Semantic-Topo-Metric Representation Guided LLM Reasoning

Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation

CityNav: Language-Goal Aerial Navigation Dataset with Geographic Information

AdaVLN: Towards Visual Language Navigation in Continuous Indoor Environments with Moving Humans

Vision and Language Navigation in the Real World via Online Visual Language Mapping

CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions

Navigation with VLM framework: Go to Any Language

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

Aerial Vision-and-Dialog Navigation

Audio Visual Language Maps for Robot Navigation

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions

Active Visual Information Gathering for Vision-Language Navigation

Visual Language Maps for Robot Navigation