Abstract:In the field of autonomous driving, Visual-and-Language Navigation (VLN) is a typical multimodal task. In the VLN task, an intelligent vehicle needs to find the target location based on user-provided navigation instructions. However, conventional VLN models generally face the problem of limited generalization ability when dealing with a large number of real-world environmental objects and language instructions. This paper proposed a novel VLN system based on large-scale pre-trained models and applied it to intelligent vehicles. The method consists of an Instruction Extraction System, a Vision-Language Association System, and a Navigational Decisions System. Specifically, a pre-trained Large Language Model (LLM) is first used to extract a series of landmark names from the user’s natural language instructions. Then, the landmark name list is inputted into a pre-trained Visual-Language Model (VLM) to infer the joint probability with environmental objects. Therefore, the image nodes that match the landmarks have been selected. Additionally, the selected image nodes are inputted into another VLM to obtain descriptions of the image nodes. Finally, LLM is used to reason navigation actions for the intelligent vehicle. With the reasoning ability of LLM, the intelligent vehicle takes navigation knowledge, visual environment descriptions, and navigation history as inputs to output navigation actions. The simulation results demonstrate that compared to other fully supervised learning methods, this approach exhibits better generalization ability in unknown environments. Based on Google Map’s Street View data, it achieves a 14.6% higher task success rate compared to the baseline model VLN Transformer.

Learning Urban Navigation via Value Iteration Network.

A LiDAR Based End to End Controller for Robot Navigation Using Deep Neural Network

Neural Network based Route Guidance Strategy in intelligent transportation systems

Highway Value Iteration Networks

A Flexible Multi-Layer Map Model Designed for Lane-Level Route Planning in Autonomous Vehicles

Real-time deep reinforcement learning based vehicle navigation

VTGNet: A Vision-based Trajectory Generation Network for Autonomous Vehicles in Urban Environments

Autonomous Navigation through intersections with Graph ConvolutionalNetworks and Conditional Imitation Learning for Self-driving Cars

Personalized Route Recommendation for Ride-Hailing with Deep Inverse Reinforcement Learning and Real-Time Traffic Conditions

Value Iteration Networks with Double Estimator for Planetary Rover Path Planning

Enhance the performance of navigation: A two-stage machine learning approach

Book Your Green Wave: Exploiting Navigation Information for Intelligent Traffic Signal Control

Integrating Deep Reinforcement Learning with Optimal Trajectory Planner for Automated Driving.

A Deep Value-network Based Approach for Multi-Driver Order Dispatching

Personalized and Context-aware Route Planning for Edge-assisted Vehicles

Personalized Route Recommendation Based on User Habits for Vehicle Navigation

Enabling Vision-and-Language Navigation for Intelligent Connected Vehicles Using Large Pre-Trained Models

Value Iteration Networks with Gated Summarization Module

Interpretable Motion Planner for Urban Driving via Hierarchical Imitation Learning

Intelligent Vehicle Decision-Making and Trajectory Planning Method Based on Deep Reinforcement Learning in the Frenet Space

An Intersection-Based Traffic Awareness Routing Protocol in VANETs Using Deep Reinforcement Learning