Abstract:This paper presents results from two machine learning components within the upstream multi-modal fusion and decision-making framework introduced previously (Garagić et al., 2018 IEEE Aerospace Conference, doi: 10.1109/AERO.2018.8396737, Garagić et al., 2019 IEEE Aerospace Conference, doi: 10.1109/AERO.2019.8742221). The first combines Bayesian change-point motion-based detection to discover new targets with online transfer learning over Convolutional Neural Networks (CNNs) to learn models of previously seen targets and adapt those models as target appearance changes. CNN detection overcomes motion-based detection problems that arise when targets stop or operate in close proximity to each other. Motion-based detection overcomes the CNN problem of initializing target appearance and adapting to target appearance changes. This novel combination of motion and CNN detectors obviates the need for prior knowledge about targets to stimulate target detection. This detection approach improves performance over that achieved without an appearance-based detector on challenging data. A closed-loop between the tracking component and the detector component modules drives detector model adaptation. The second component combines multi-sensor, multi-modality features from upstream detection and tracking components to learn dynamic compact probabilistic representations of targets as manifolds in a joint space over all inputs. This Multi-Modal Deep Belief Network (MMDBN) approach incorporates supervised deep autoencoders and unsupervised Dirichlet process mixture models. Supervision within MMDBN arises from consistent associations between sensor and modality inputs for a target of interest. MMDBN thus learns and optimally leverages inter-sensor, inter-modality features of targets. This approach obviates the need for explicit, hand-crafted intermodal features because it implicitly learns both inter-modal features and a likelihood model to evaluate those features. This fusion approach improves tracking performance over a traditional fusion approach on challenging data. This overall approach offers multiple benefits in addition to outperforming single-modality and traditional fusion approaches. It is feature agnostic by accommodating any data source that provides a feature set of pre-determined dimensionality for each target at each time step. This paper describes novel methods for enhancing motion-based tracking by adding an appearance-based sidecar that provides to improve DTC performance in difficult conditions. It also showed the benefits of a feature-level fusion approach in performing DTC on several challenging multi-modality, multi-platforms AFRL ESCAPE data scenarios.

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Object Detection Using Multi-Sensor Fusion Based on Deep Learning

Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

Machine Learning Multi-Modality Fusion Approaches Outperform Single-Modality & Traditional Approaches

Enabling Efficient Deep Convolutional Neural Network-based Sensor Fusion for Autonomous Driving

mmFUSION: Multimodal Fusion for 3D Objects Detection

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

RGB-LiDAR fusion for accurate 2D and 3D object detection

A Generalized Multi-Modal Fusion Detection Framework

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Object detection using depth completion and camera-LiDAR fusion for autonomous driving

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Transformer-Based Optimized Multimodal Fusion for 3D Object Detection in Autonomous Driving

${\mathsf{EZFusion}}$: A Close Look at the Integration of LiDAR, Millimeter-Wave Radar, and Camera for Accurate 3D Object Detection and Tracking

Real-Time Hybrid Multi-Sensor Fusion Framework for Perception in Autonomous Vehicles