Abstract:With the popularity of online learning and due to the significant influence of emotion on the learning effect, more and more researches focus on emotion recognition in online learning. Most of the current research uses the comments of the learning platform or the learner’s expression for emotion recognition. The research data on other modalities are scarce. Most of the studies also ignore the impact of instructional videos on learners and the guidance of knowledge on data. Because of the need for other modal research data, we construct a synchronous multimodal data set for analyzing learners’ emotional states in online learning scenarios. The data set recorded the eye movement data and photoplethysmography (PPG) signals of 68 subjects and the instructional video they watched. For the problem of ignoring the instructional videos on learners and ignoring the knowledge, a multimodal emotion recognition method in video learning based on knowledge enhancement is proposed. This method uses the knowledge-based features extracted from instructional videos, such as brightness, hue, saturation, the videos’ click-through rate, and emotion generation time, to guide the emotion recognition process of physiological signals. This method uses Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to extract deeper emotional representation and spatiotemporal information from shallow features. The model uses multi-head attention (MHA) mechanism to obtain critical information in the extracted deep features. Then, Temporal Convolutional Network (TCN) is used to learn the information in the deep features and knowledge-based features. Knowledge-based features are used to supplement and enhance the deep features of physiological signals. Finally, the fully connected layer is used for emotion recognition, and the recognition accuracy reaches 97.51%. Compared with two recent researches, the accuracy improved by 8.57% and 2.11%, respectively. On the four public data sets, our proposed method also achieves better results compared with the two recent researches. The experiment results show that the proposed multimodal emotion recognition method based on knowledge enhancement has good performance and robustness.

Multimodal Physiological Signals Fusion for Online Emotion Recognition

Online multi-hypergraph fusion learning for cross-subject emotion recognition

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

A multi-stage dynamical fusion network for multimodal emotion recognition

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition

Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Multimodal Latent Emotion Recognition from Micro-expression and Physiological Signals

Updating recommendations for prevention of coronary heart disease.

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

A Method of Multimodal Emotion Recognition in Video Learning Based on Knowledge Enhancement

Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data

Hierarchical fusion of visual and physiological signals for emotion recognition

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals

Multimodal Emotion Recognition Model using Physiological Signals