Abstract:Sentiment analysis can be used to study an individual or a group’s emotions and attitudes towards other people and entities like products, services, or social events. With the advancements in the field of deep learning, the enormity of available information on internet, chiefly on social media, combined with powerful computing machines, it’s just a matter of time before artificial intelligence (AI) systems make their presence in every aspect of human life, making our lives more introspective. In this paper, we propose to implement a multimodal sentiment prediction system that can analyze the emotions predicted from different modal sources such as video, audio and text and integrate them to recognize the group emotions of the students in a classroom. Our experimental setup involves a digital video camera with microphones to capture the live video and audio feeds of the students during a lecture. The students are advised to provide their digital feedback on the lecture as ‘tweets’ on their twitter account addressed to the lecturer’s official twitter account. The audio and video frames are separated from the live streaming video using tools such as lame and ffmpeg. A twitter API was used to access and extract messages from twitter platform. The audio and video features are extracted using Mel-Frequency Cepstral Co-efficients (MFCC) and Haar Cascades classifier respectively. The extracted features are then passed to the Convolutional Neural Network (CNN) model trained on the FER2013 facial images database to generate the feature vector for classification of video-based emotions. A Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM), trained on speech emotion corpus database was used to train on the audio features. A lexicon-based approach with senti-word dictionary and learning based approach with custom dataset trained by Support Vector Machines (SVM) was used in the twitter-texts based approach. A decision-level fusion algorithm was applied on these three different modal schemes to integrate the classification results and deduce the overall group emotions of the students. The use-case of this proposed system will be in student emotion recognition, employee performance feedback, monitoring or surveillance-based systems. The implemented system framework was tested in a classroom environment during a live lecture and the predicted emotions demonstrated the classification accuracy of our approach.

Multimodal Emotion Recognition and State Analysis of Classroom Video and Audio Based on Deep Neural Network

Multimodal Emotional Classification Based on Meaningful Learning

Classroom Student Emotions Classification from Facial Expressions and Speech Signals using Deep Learning

Student's Emotion Recognition using Multimodality and Deep Learning

A Method for Analyzing Learning Sentiment Based on Classroom Time-Series Images

Advances and Application of Facial Expression and Learning Emotion Recognition in Classroom

Multimodal biometric fusion sentiment analysis of teachers and students based on classroom surveillance video streaming

Emotion recognition and achievement prediction for foreign language learners under the background of network teaching

Student Sentiment Analysis and Classroom Feedback Prediction Using Deep Learning

Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network

A facial and vocal emotion recognition system based on deep learning

Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios

A Method of Multimodal Emotion Recognition in Video Learning Based on Knowledge Enhancement

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Exploring multimodal data analysis for emotion recognition in teachers’ teaching behavior based on LSTM and MSCNN

Multimodal Decision-level Group Sentiment Prediction of Students in Classrooms

Analysis of English classroom teaching behavior and strategies under adaptive deep learning under cognitive psychology

Research on Classroom Emotion Recognition Algorithm Based on Visual Emotion Classification

Realization of Self-Adaptive Higher Teaching Management Based Upon Expression and Speech Multimodal Emotion Recognition

Evaluation of Online Teaching Quality Based on Facial Expression Recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG