Abstract:In the post-COVID-19 era, online learning has changed from an emergency teaching method to a new, normalized one. However, online learning is often plagued by low participation and high dropout compared to offline learning. A critical way to address these issues is the accurate detection of student engagement, which will help teachers promptly assess learners' status. Image data are one of the most straightforward ways to reflect student engagement levels. However, traditional engagement detection methods with images either rely on manual analysis or interfere with student behavior, which leads to a need for more objectivity in final engagement levels. This paper proposes a system that utilizes images obtained from individual webcams in online classrooms. Based on the techniques of multi-dimensional feature fusion and multimodal analysis, this system can rapidly detect and output students' classroom engagement levels which provides real-time support for teachers to adjust their teaching methods during the teaching process, aiming to enhance students' engagement in online courses. In the feature extraction module, VGG16 is utilized to recognize students' facial expressions, ResNet-101 is designed to estimate head pose in each image, and Mediapipe is applied to estimate facial landmarks that reflect eye–mouth behavior. Subsequently, a BP neural network is constructed to fuse these multi-dimensional features and output the engagement level in each image in the data fusion module. The present method is evaluated on the wacv2016 data set and achieves an accuracy of 62.03%, outperforming the single-dimensional method. It is also applied in online courses to demonstrate its validity in the actual scenario further. Pearson Correlation between engagement levels calculated by our multi-dimensional fusion method and NSSE–China survey scores filled out by students is 0.714. It indicates that the method can enable real-time monitoring of students' classroom engagement with similar results to traditional questionnaires with little human resources and time.

Engagement Detection in Online Learning Based on Pre-trained Vision Transformer and Temporal Convolutional Network

Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network

Student engagement detection in online environment using computer vision and multi-dimensional feature fusion

TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals

Exploiting Temporal Coherence for Self-Supervised Visual Tracking by Using Vision Transformer

An Online Education Effect Assessing Method Based on YOLOv8 and Vision Transformer

Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks

LS-VIT: Vision Transformer for action recognition based on long and short-term temporal difference

Class-attention video transformer for engagement prediction

Detecting disengagement in virtual learning as an anomaly using temporal convolutional network autoencoder

DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement Estimation in Conversation

Temporally Efficient Vision Transformer for Video Instance Segmentation

Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Advanced Multi-Instance Learning Method with Multi-features Engineering and Conservative Optimization for Engagement Intensity Prediction

Class-attention Video Transformer for Engagement Intensity Prediction

DCT-net: A Deep Co-Interactive Transformer Network for Video Temporal Grounding

Deep Recurrent Multi-instance Learning with Spatio-temporal Features for Engagement Intensity Prediction

DAT++: Spatially Dynamic Vision Transformer with Deformable Attention

DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition