Abstract:With the popularity of online learning and due to the significant influence of emotion on the learning effect, more and more researches focus on emotion recognition in online learning. Most of the current research uses the comments of the learning platform or the learner’s expression for emotion recognition. The research data on other modalities are scarce. Most of the studies also ignore the impact of instructional videos on learners and the guidance of knowledge on data. Because of the need for other modal research data, we construct a synchronous multimodal data set for analyzing learners’ emotional states in online learning scenarios. The data set recorded the eye movement data and photoplethysmography (PPG) signals of 68 subjects and the instructional video they watched. For the problem of ignoring the instructional videos on learners and ignoring the knowledge, a multimodal emotion recognition method in video learning based on knowledge enhancement is proposed. This method uses the knowledge-based features extracted from instructional videos, such as brightness, hue, saturation, the videos’ click-through rate, and emotion generation time, to guide the emotion recognition process of physiological signals. This method uses Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to extract deeper emotional representation and spatiotemporal information from shallow features. The model uses multi-head attention (MHA) mechanism to obtain critical information in the extracted deep features. Then, Temporal Convolutional Network (TCN) is used to learn the information in the deep features and knowledge-based features. Knowledge-based features are used to supplement and enhance the deep features of physiological signals. Finally, the fully connected layer is used for emotion recognition, and the recognition accuracy reaches 97.51%. Compared with two recent researches, the accuracy improved by 8.57% and 2.11%, respectively. On the four public data sets, our proposed method also achieves better results compared with the two recent researches. The experiment results show that the proposed multimodal emotion recognition method based on knowledge enhancement has good performance and robustness.

Exploring multimodal data analysis for emotion recognition in teachers’ teaching behavior based on LSTM and MSCNN

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Emotion recognition and achievement prediction for foreign language learners under the background of network teaching

Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios

Multimodal Emotion Recognition and State Analysis of Classroom Video and Audio Based on Deep Neural Network

A Method of Multimodal Emotion Recognition in Video Learning Based on Knowledge Enhancement

Analysis of English classroom teaching behavior and strategies under adaptive deep learning under cognitive psychology

Multimodal biometric fusion sentiment analysis of teachers and students based on classroom surveillance video streaming

Realization of Self-Adaptive Higher Teaching Management Based Upon Expression and Speech Multimodal Emotion Recognition

Emotion Analysis Method of Teaching Evaluation Texts Based on Deep Learning in Big Data Environment

A Method for Analyzing Learning Sentiment Based on Classroom Time-Series Images

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

A structural model of teachers’ teaching competencies based on multimodal affective features of support vector machines

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Constructing a Multimodal Music Teaching Model in College by Integrating Emotions

A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis

Design of Blended Teaching Model Based on Emotion Recognition and Language Learning

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Multimodal interaction enhanced representation learning for video emotion recognition

Construction and Evaluation of College English Translation Teaching Model Based on Multimodal Integration