Abstract:With the popularity of online learning and due to the significant influence of emotion on the learning effect, more and more researches focus on emotion recognition in online learning. Most of the current research uses the comments of the learning platform or the learner’s expression for emotion recognition. The research data on other modalities are scarce. Most of the studies also ignore the impact of instructional videos on learners and the guidance of knowledge on data. Because of the need for other modal research data, we construct a synchronous multimodal data set for analyzing learners’ emotional states in online learning scenarios. The data set recorded the eye movement data and photoplethysmography (PPG) signals of 68 subjects and the instructional video they watched. For the problem of ignoring the instructional videos on learners and ignoring the knowledge, a multimodal emotion recognition method in video learning based on knowledge enhancement is proposed. This method uses the knowledge-based features extracted from instructional videos, such as brightness, hue, saturation, the videos’ click-through rate, and emotion generation time, to guide the emotion recognition process of physiological signals. This method uses Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to extract deeper emotional representation and spatiotemporal information from shallow features. The model uses multi-head attention (MHA) mechanism to obtain critical information in the extracted deep features. Then, Temporal Convolutional Network (TCN) is used to learn the information in the deep features and knowledge-based features. Knowledge-based features are used to supplement and enhance the deep features of physiological signals. Finally, the fully connected layer is used for emotion recognition, and the recognition accuracy reaches 97.51%. Compared with two recent researches, the accuracy improved by 8.57% and 2.11%, respectively. On the four public data sets, our proposed method also achieves better results compared with the two recent researches. The experiment results show that the proposed multimodal emotion recognition method based on knowledge enhancement has good performance and robustness.

Multi-Modal Emotion Recognition for Online Education Using Emoji Prompts

Emotion Detection in Online Social Networks: A Multilabel Learning Approach

SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering

A Multimodal Intelligent Emotion Perception Framework by Data-driven and Knowledge-guided

Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios

A Method for Analyzing Learning Sentiment Based on Classroom Time-Series Images

Emotion generation method in online physical education teaching based on data mining of teacher-student interactions

A Method of Multimodal Emotion Recognition in Video Learning Based on Knowledge Enhancement

Research on Image-text Multimodal Emotions Analysis with Fused Emoji

Multimodal Emotion Recognition and State Analysis of Classroom Video and Audio Based on Deep Neural Network

Emotion Dictionary Learning with Modality Attentions for Mixed Emotion Exploration

EmoChat: Bringing Multimodal Emotion Detection to Mobile Conversation

Using Emoji as an Emotion Modality in Text-Based Depression Detection

Design of an Online Education Evaluation System Based on Multimodal Data of Learners.

Research on MOOC Reviews Oriented Sentiment Analysis by Awareness of Emotional Distinctions

Optimizing Interactive Online Education Through Advanced Multimodal Emotion Recognition Techniques

Multimodal Sentiment Analysis of Graphic Texts Based on Multicategorical Relative Fusion

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Advances and Application of Facial Expression and Learning Emotion Recognition in Classroom

PEDM: A Multi-task Learning Model for Persona-aware Emoji-embedded Dialogue Generation

Emoticon Analysis for Chinese Social Media and E-commerce: the AZEmo System.