Abstract:In the post-COVID-19 era, online learning has changed from an emergency teaching method to a new, normalized one. However, online learning is often plagued by low participation and high dropout compared to offline learning. A critical way to address these issues is the accurate detection of student engagement, which will help teachers promptly assess learners' status. Image data are one of the most straightforward ways to reflect student engagement levels. However, traditional engagement detection methods with images either rely on manual analysis or interfere with student behavior, which leads to a need for more objectivity in final engagement levels. This paper proposes a system that utilizes images obtained from individual webcams in online classrooms. Based on the techniques of multi-dimensional feature fusion and multimodal analysis, this system can rapidly detect and output students' classroom engagement levels which provides real-time support for teachers to adjust their teaching methods during the teaching process, aiming to enhance students' engagement in online courses. In the feature extraction module, VGG16 is utilized to recognize students' facial expressions, ResNet-101 is designed to estimate head pose in each image, and Mediapipe is applied to estimate facial landmarks that reflect eye–mouth behavior. Subsequently, a BP neural network is constructed to fuse these multi-dimensional features and output the engagement level in each image in the data fusion module. The present method is evaluated on the wacv2016 data set and achieves an accuracy of 62.03%, outperforming the single-dimensional method. It is also applied in online courses to demonstrate its validity in the actual scenario further. Pearson Correlation between engagement levels calculated by our multi-dimensional fusion method and NSSE–China survey scores filled out by students is 0.714. It indicates that the method can enable real-time monitoring of students' classroom engagement with similar results to traditional questionnaires with little human resources and time.

Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

CNN for Facial Emotion Recognition in Online Learning Platforms to Identify Learner Engagement

Enhancing Student Engagement in Online Learning through Facial Expression Analysis and Complex Emotion Recognition using Deep Learning

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model

A Deep Learning-Based Approach for Adaptive Virtual Learning with Human Facial Emotion Detection

An optimized deep convolutional neural network for adaptive learning using feature fusion in multimodal data

Application of Deep Learning on Student Engagement in e-learning environments

An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition

A Deep Learning-based Emotion Recognition System for Interactive E-Learning

Implementation of an adaptive E-learning platform with facial emotion recognition

Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network

Exploring AI Facial Recognition for Real-time Emotion Detection: Assessing Student Engagement in Online Learning Environments

Multimodal Engagement Recognition From Image Traits Using Deep Learning Techniques

Artificial intelligence based cognitive state prediction in an e-learning environment using multimodal data

Student engagement detection in online environment using computer vision and multi-dimensional feature fusion

Classroom Student Emotions Classification from Facial Expressions and Speech Signals using Deep Learning

A Deep Learning Approach for Online Learning Emotion Recognition

Automatic Recognition and Analysis of Academic Emotions Based on Facial Expressions during Online Learning Environments

Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning

Real Time Emotion Analysis Using Deep Learning for Education, Entertainment, and Beyond