Abstract:Autonomous vehicle navigation is a key challenge in artificial intelligence, requiring robust and accurate decision-making processes. This research introduces a new end-to-end method that exploits multimodal information from a single monocular camera to improve the steering predictions for self-driving cars. Unlike conventional models that require several sensors which can be costly and complex or rely exclusively on RGB images that may not be robust enough under different conditions, our model significantly improves vehicle steering prediction performance from a single visual sensor. By focusing on the fusion of RGB imagery with depth completion information or optical flow data, we propose a comprehensive framework that integrates these modalities through both early and hybrid fusion techniques. We use three distinct neural network models to implement our approach: Convolution Neural Network - Neutral Circuit Policy (CNN-NCP) , Variational Auto Encoder - Long Short-Term Memory (VAE-LSTM) , and Neural Circuit Policy architecture VAE-NCP. By incorporating optical flow into the decision-making process, our method significantly advances autonomous navigation. Empirical results from our comparative study using Boston driving data show that our model, which integrates image and motion information, is robust and reliable. It outperforms state-of-the-art approaches that do not use optical flow, reducing the steering estimation error by 31%. This demonstrates the potential of optical flow data, combined with advanced neural network architectures (a CNN-based structure for fusing data and a Recurrence-based network for inferring a command from latent space), to enhance the performance of autonomous vehicles steering estimation.

Optical Flow as a Virtual Gyro and Altimeter for Safer Spacecraft Visual Inertial Target Approaches

An Efficient Gyro-Aided Optical Flow Estimation in Fast Rotations with Auto-Calibration

Dynamic Illumination Optical Flow Computing for Sensing Multiple Mobile Robots from a Drone.

Optical Flow Based Obstacle Detection in Static Environment

Improving Optical Flow Sensor Using a Gimbal for Quadrotor Navigation in GPS-Denied Environment

A Homography-Based Visual Inertial Fusion Method for Robust Sensing of a Micro Aerial Vehicle

Movement Tracking by Optical Flow Assisted Inertial Navigation

Uncertainty Estimation of Dense Optical Flow for Robust Visual Navigation

Vision-based Target Tracking for UAV and Relative Depth Estimation using Optical Flow

Uncertainty Estimation of Dense Optical-Flow for Robust Visual Navigation

Incremental Nonlinear Dynamic Inversion based Optical Flow Control for Flying Robots: An Efficient Data-driven Approach

Fusing Stereopsis & Corner Sparse Optical Flow for Real-time Obstacle Avoidance of Unmanned Aerial Vehicle

Distance estimation with efference copies and optical flow maneuvers: a stability-based strategy

Seeing Through Pixel Motion: Learning Obstacle Avoidance from Optical Flow with One Camera

Metric Sensing and Control of a Quadrotor Using a Homography-Based Visual Inertial Fusion Method.

Optical flow-based control for micro air vehicles: an efficient data-driven incremental nonlinear dynamic inversion approach

Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering

Visual Navigation Using Sparse Optical Flow and Time-to-Transit

Optical flow inversion for remote sensing image dense registration and sensor's attitude motion high-accurate measurement

Optical Flow-Based Obstacle Detection for Mid-Air Collision Avoidance

Vertical Landing for Micro Air Vehicles using Event-Based Optical Flow