Abstract:Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Multimodal Emotion Recognition Method Based on Convolutional Auto-Encoder.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear pooling

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

Multimodal emotion recognition based on manifold learning and convolution neural network

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Multimodal Emotion Recognition Model using Physiological Signals

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Emotion Recognition from EEG Signal Using CA-GCN

Multimodal Emotion Recognition Using Different Fusion Techniques

Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multi-Scale Masked Autoencoders for Cross-Session Emotion Recognition

An improved multi-input deep convolutional neural network for automatic emotion recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks