Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

Cloud Recognition Using Multimodal Information: A Review

Emotion Recognition in Videos via Fusing Multimodal Features.

Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks.

Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network

Cloud Type Classification Using Multi-modal Information Based on Multi-task Learning

Multi-View Ground-Based Cloud Recognition by Transferring Deep Visual Information

Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

Multimodal Ground-Based Remote Sensing Cloud Classification Via Learning Heterogeneous Deep Features.

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

A Review of Multimodal Medical Image Fusion Techniques

A Survey on Deep Learning for Multimodal Data Fusion

Multimodal Sentiment Analysis Based on Fusion Methods: A Survey

Deep Multimodal Data Fusion

Automatic cloud detection method for optical remote sensing images combining channel attention mechanism and multi-feature fusion

Cloud Detection Using Super Pixel Classification and Semantic Segmentation

Multimodal Emotion Recognition Based on Feature Fusion.

RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Deep Tensor Fusion Network for Multimodal Ground-Based Cloud Classification in Weather Station Networks

The Fusion Strategy of 2D and 3D Information Based on Deep Learning: A Review