Abstract:The authors propose a deep learning‐based sensor fusion framework that uses both camera and LiDAR sensors in a multi‐modal and multi‐view setting. In order to leverage both data streams, two fusion mechanisms are incorporated: element‐wise multiplication and multi‐modal factorised bilinear pooling. The authors provide a detailed study of important design choices that contribute to the performance of deep learning‐based sensor fusion frameworks such as data augmentation, multi‐task learning, and the design of convolutional architecture. Perception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

Incremental Learning with Balanced Update on Receptive Fields for Multi-Sensor Data Fusion

A Fusion Method with Incremental Learning Ability

Texture Aware Learning-based Image Fusion Method for Fixed Focal-length Cameras

Incremental learning method for unified camera calibration

Multi-Sensors Data Tracking Fusion Based on a Neural Network Filter

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Boosting Image Super-Resolution Via Fusion of Complementary Information Captured by Multi-Modal Sensors

MSFusion: Multilayer Sensor Fusion-Based Robust Motion Estimation

Incremental learning of LSTM framework for sensor fusion in attitude estimation

Enhanced Radar Perception via Multi-Task Learning: Towards Refined Data for Sensor Fusion Applications

A Novel Multi-Sensor Hybrid Fusion Framework

Lvio-Fusion: A Self-adaptive Multi-sensor Fusion SLAM Framework Using Actor-critic Method

A Robust Fusion Algorithm For Multi-Sensor Tracking

A New Adaptive Weighted Fusion Algorithm for Multi-Sensor Tracking

Learning Selective Sensor Fusion for States Estimation

Multi-Sensor Image Fusion Using Optimized Support Vector Machine and Multiscale Weighted Principal Component Analysis

A Multi-sensor Fusion Algorithm with Feedback Based on Fuzzy C-Means and Maximum Entropy Principle

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

Multi-Parameter Based Self-Feedback Effectiveness Evaluation in a Multi-Sensor Fusion Positioning System.

Different Input Resolutions and Arbitrary Output Resolution: A Meta Learning-Based Deep Framework for Infrared and Visible Image Fusion

A new method for asynchronous multisensor information fusion