Abstract:Emotion understanding represents a core aspect of human communication. Our social behaviours are closely linked to expressing our emotions and understanding others’ emotional and mental states through social signals. The majority of the existing work proceeds by extracting meaningful features from each modality and applying fusion techniques either at a feature level or decision level. However, these techniques are incapable of translating the constant talk and feedback between different modalities. Such constant talk is particularly important in continuous emotion recognition, where one modality can predict, enhance and complement the other. This article proposes three multisensory integration models, based on different pathways of multisensory integration in the brain; that is, integration by convergence, early cross-modal enhancement, and integration through neural synchrony. The proposed models are designed and implemented using third-generation neural networks, Spiking Neural Networks (SNN). The models are evaluated using widely adopted, third-party datasets and compared to state-of-the-art multimodal fusion techniques, such as early, late and deep learning fusion. Evaluation results show that the three proposed models have achieved comparable results to the state-of-the-art supervised learning techniques. More importantly, this article demonstrates plausible ways to translate constant talk between modalities during the training phase, which also brings advantages in generalisation and robustness to noise.

Neural Dependency Coding inspired Multimodal Fusion

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Progressive Fusion for Multimodal Integration

Improving Multimodal fusion via Mutual Dependency Maximisation

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Brain-inspired Multimodal Learning Based on Neural Networks

Attention Bottlenecks for Multimodal Fusion

Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling

Deep Multimodal Data Fusion

Multimodal Multi-loss Fusion Network for Sentiment Analysis

Multimodal Language Analysis with Recurrent Multistage Fusion

Adaptive Fusion Techniques for Multimodal Data

A Cross-Modal Fusion Network Based on Self-Attention and Residual Structure for Multimodal Emotion Recognition

Image and Encoded Text Fusion for Multi-Modal Classification

Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data

Interpretation on Multi-modal Visual Fusion

NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Quantum-inspired multimodal fusion for video sentiment analysis