Abstract:This paper presents results from two machine learning components within the upstream multi-modal fusion and decision-making framework introduced previously (Garagić et al., 2018 IEEE Aerospace Conference, doi: 10.1109/AERO.2018.8396737, Garagić et al., 2019 IEEE Aerospace Conference, doi: 10.1109/AERO.2019.8742221). The first combines Bayesian change-point motion-based detection to discover new targets with online transfer learning over Convolutional Neural Networks (CNNs) to learn models of previously seen targets and adapt those models as target appearance changes. CNN detection overcomes motion-based detection problems that arise when targets stop or operate in close proximity to each other. Motion-based detection overcomes the CNN problem of initializing target appearance and adapting to target appearance changes. This novel combination of motion and CNN detectors obviates the need for prior knowledge about targets to stimulate target detection. This detection approach improves performance over that achieved without an appearance-based detector on challenging data. A closed-loop between the tracking component and the detector component modules drives detector model adaptation. The second component combines multi-sensor, multi-modality features from upstream detection and tracking components to learn dynamic compact probabilistic representations of targets as manifolds in a joint space over all inputs. This Multi-Modal Deep Belief Network (MMDBN) approach incorporates supervised deep autoencoders and unsupervised Dirichlet process mixture models. Supervision within MMDBN arises from consistent associations between sensor and modality inputs for a target of interest. MMDBN thus learns and optimally leverages inter-sensor, inter-modality features of targets. This approach obviates the need for explicit, hand-crafted intermodal features because it implicitly learns both inter-modal features and a likelihood model to evaluate those features. This fusion approach improves tracking performance over a traditional fusion approach on challenging data. This overall approach offers multiple benefits in addition to outperforming single-modality and traditional fusion approaches. It is feature agnostic by accommodating any data source that provides a feature set of pre-determined dimensionality for each target at each time step. This paper describes novel methods for enhancing motion-based tracking by adding an appearance-based sidecar that provides to improve DTC performance in difficult conditions. It also showed the benefits of a feature-level fusion approach in performing DTC on several challenging multi-modality, multi-platforms AFRL ESCAPE data scenarios.

A Multi-Layer K-means Approach for Multi-Sensor Data Pattern Recognition in Multi-Target Localization

Graph Based Multi-Layer K-Means++ (G-MLKM) for Sensory Pattern Analysis in Constrained Spaces

MultiRobot Precise Localization Based on Multisensor Fusion.

Data Association in Multi-Target Localization Using Bearing-Only Sensor Networks

Multi-Sensors Data Tracking Fusion Based on a Neural Network Filter

Detection, tracking and recognition of multi-targets using fusion data from different sensors

Multilevel Structure Extraction-Based Multi-Sensor Data Fusion

Clustered Multi-Task Learning for Automatic Radar Target Recognition.

A Multi-Sensor Fusion Framework for Localization Using LiDAR, IMU and RGB-DCamera

Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving

Machine Learning Multi-Modality Fusion Approaches Outperform Single-Modality & Traditional Approaches

Multi-sensor fusion for robust localization with moving object segmentation in complex dynamic 3D scenes

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

A Multimodal Data Harness Approach of Mobile Sensors Trajectory planning for Target Tracking

Adaptive Human-in-the-loop Multi-Target Recognition Improved by Learning

A multi-class classification approach for target localization in wireless sensor networks

Multi-vehicle Detection and Tracking Based on Kalman Filter and Data Association

A Novel Fuzzy Pattern Recognition Data Association Method for Biased Sensor Data.

Research on multi-target tracking method based on multi-sensor fusion

Research on Spatial Target Recognition Method Based on Multi-source Sensor Fusion

Multi-object Detection, Tracking and Prediction in Rugged Dynamic Environments