Abstract:Introduction: Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. Methods: In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network–time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. Results: To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31–50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. Discussion: The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Brain fMRI segmentation under emotion stimuli incorporating attention-based deep convolutional neural networks

Automated 3D Fetal Brain Segmentation Using an Optimized Deep Learning Approach

DISTINGUISHING BIPOLAR DEPRESSION FROM MAJOR DEPRESSIVE DISORDER USING FNIRS AND DEEP NEURAL NETWORK

Emotional Brain State Classification on fMRI Data Using Deep Residual and Convolutional Networks

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Cross-Subject Emotion Recognition Brain–Computer Interface Based on fNIRS and DBJNet

FBN-TCN: Temporal convolutional neural network based on spatial domain fusion brain networks for affective brain–computer interfaces

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Exploring Emotions in EEG: Deep Learning Approach with Feature Fusion

Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signals

Multi-domain fusion deep graph convolution neural network for EEG emotion recognition

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

MAS-DGAT-Net: A dynamic graph attention network with multibranch feature extraction and staged fusion for EEG emotion recognition

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Emotion recognition with convolutional neural network and EEG-based EFDMs

Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition

Analysis of Psychological and Emotional Tendency Based on Brain Functional Imaging and Deep Learning

Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

EEG emotion recognition based on data-driven signal auto-segmentation and feature fusion

Deep Learning-Based Multiclass Brain Tissue Segmentation in Fetal MRIs