Abstract:Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest. Most research has focused on ground-based agents, while UAV-based VLN remains relatively underexplored. Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navigation tasks between ground and aerial environments. To address these disparities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology. To enable realistic UAV trajectory simulation in VLN tasks, we propose the OpenUAV platform, which features diverse environments, realistic flight control, and extensive algorithmic support. We further construct a target-oriented VLN dataset consisting of approximately 12k trajectories on this platform, serving as the first dataset specifically designed for realistic UAV VLN tasks. To tackle the challenges posed by complex aerial environments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help, which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks. We also propose a UAV navigation LLM that, given multi-view images, task descriptions, and assistant instructions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajectory generation. The evaluation results of our method significantly outperform the baseline models, while there remains a considerable gap between our results and those achieved by human operators, underscoring the challenge presented by the UAV-Need-Help task.

Learning Visual Navigation System in Simulation for Autonomous Ground Vehicles in Real World

A LiDAR Based End to End Controller for Robot Navigation Using Deep Neural Network

ColAG: A Collaborative Air-Ground Framework for Perception-Limited UGVs' Navigation

End-to-End UAV Simulation for Visual SLAM and Navigation

Learning to navigate efficiently and precisely in real environments

Vision and Language Navigation in the Real World via Online Visual Language Mapping

Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation

Autonomous Navigation with Mobile Robots Using Deep Learning and the Robot Operating System

An Intelligent Navigation Control Approach for Autonomous Unmanned Vehicles via Deep Learning-Enhanced Visual SLAM Framework

BADGR: An Autonomous Self-Supervised Learning-Based Navigation System

Navigating Unstructured Space: Deep Action Learning-Based Obstacle Avoidance System for Indoor Automated Guided Vehicles

Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey

UAS Navigation in the Real World Using Visual Observation

Using Deep Reinforcement Learning with Automatic Curriculum Learning for Mapless Navigation in Intralogistics

Deep Reinforcement Learning with Heuristic Corrections for UGV Navigation

Sim-to-Real: Mapless Navigation for USVs Using Deep Reinforcement Learning

Navigating to objects in the real world

BEVNav: Robot Autonomous Navigation Via Spatial-Temporal Contrastive Learning in Bird's-Eye View

Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics

Learning to Navigate from Simulation via Spatial and Semantic Information Synthesis with Noise Model Embedding

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology