Abstract:Multimodal learning has been lacking principled ways of combining information from different modalities and learning a low-dimensional manifold of meaningful representations. We study multimodal learning and sensor fusion from a latent variable perspective. We first present a regularized recurrent attention filter for sensor fusion. This algorithm can dynamically combine information from different types of sensors in a sequential decision making task. Each sensor is bonded with a modular neural network to maximize utility of its own information. A gating modular neural network dynamically generates a set of mixing weights for outputs from sensor networks by balancing utility of all sensors' information. We design a co-learning mechanism to encourage co-adaption and independent learning of each sensor at the same time, and propose a regularization based co-learning method. In the second part, we focus on recovering the manifold of latent representation. We propose a co-learning approach using probabilistic graphical model which imposes a structural prior on the generative model: multimodal variational RNN (MVRNN) model, and derive a variational lower bound for its objective functions. In the third part, we extend the siamese structure to sensor fusion for robust acoustic event detection. We perform experiments to investigate the latent representations that are extracted; works will be done in the following months. Our experiments show that the recurrent attention filter can dynamically combine different sensor inputs according to the information carried in the inputs. We consider MVRNN can identify latent representations that are useful for many downstream tasks such as speech synthesis, activity recognition, and control and planning. Both algorithms are general frameworks which can be applied to other tasks where different types of sensors are jointly used for decision making.

Multimodal sensor fusion in the latent representation space

Latent Variable Algorithms for Multimodal Learning and Sensor Fusion

Robust Multi-Modal Sensor Fusion: An Adversarial Approach

Dual Low-Rank Multimodal Fusion

Multimodal Data Fusion in High-Dimensional Heterogeneous Datasets via Generative Models

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Latent Representation Learning Model for Multi-Band Images Fusion via Low-Rank and Sparse Embedding

Deep Fusion Of Heterogeneous Sensor Data

Deep Multimodal Data Fusion

Multimodal Data Fusion Using Non-Sparse Multi-Kernel Learning With Regularized Label Softening

Multimodal Fusion Method Based on Self-Attention Mechanism

Weakly Paired Multimodal Fusion for Object Recognition.

Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data

Multimodal Remote Sensing Data Classification Based on Gaussian Mixture Variational Dynamic Fusion Network

Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association

Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences

Heterogeneous Sensor Data Fusion by Deep Multimodal Encoding

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

SFusion: Self-attention based N-to-One Multimodal Fusion Block

Incomplete Multimodal Learning for Remote Sensing Data Fusion