Abstract:Introduction: Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. Methods: In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network–time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. Results: To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31–50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. Discussion: The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling

Emotion Recognition in Videos via Fusing Multimodal Features.

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

Multimodal Emotion Recognition Model using Physiological Signals

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Multimodal emotion recognition based on manifold learning and convolution neural network

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Heterogeneous Convolutional Neural Networks for Emotion Recognition Combined with Multimodal Factorised Bilinear Pooling and Mobile Application Recommendation

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition

Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

A multi-stage dynamical fusion network for multimodal emotion recognition

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Feature-level fusion of multimodal physiological signals for emotion recognition

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition