Abstract:It is crucial to evaluate learning outcomes by identifying the cognitive state of the learner during the learning process. Studies utilizing Electroencephalography (EEG) and other peripheral physiological signals, combined with deep learning models, have demonstrated improved performance in cognitive state recognition. These studies have primarily focused on unimodal data, which are vulnerable to various types of noise, making it difficult to fully capture and represent cognitive states. Leveraging the complementarity between multimodal physiological signals can mitigate the impact of anomalies in unimodal data, thereby improving the accuracy and stability of cognitive state recognition. Therefore, this study proposes a multimodal physiological signal feature representation fusion model based on multi-level attention (PSFMMA). The model aims to integrate multimodal physiological signals to identify learners' cognitive states with greater stability and accuracy. PSFMMA first extracts the temporal features of physiological signals by multiplexing the embedding layer. Subsequently, it generates signal representation vectors by further extracting semantic features through a signal feature mapping layer and enhancing important features with designed attention modules. Finally, the model employs an attention mechanism based on different signal representation vectors to fuse multimodal information for identifying learners' cognitive states. This study designs various learning activities and collects electroencephalography (EEG), electrodermal activity (EDA), and photoplethysmography (PPG) data from 22 participants engaging in these activities to create the Based on Learning Activities Collection (BLAC) dataset. The proposed model was evaluated on the BLAC dataset, achieving an identification accuracy of 96.32 ± 0.32%. The results demonstrate that the model can effectively recognize learners' cognitive states. Furthermore, the model's performance was validated on the publicly available emotion classification dataset DEAP, attaining an accuracy of 99.15 ± 0.12%. The source code is available at https://github.com/chengshudaxuesheng/PSFMMA.

Collaborative Learning Quality Classification Through Physiological Synchrony Recorded by Wearable Biosensors

Exploring What Synchronized Physiological Arousal Can Reveal about the Social Regulatory Process in a Collaborative Argumentation Activity.

Brain-to-brain Synchrony Predicts Classroom Performance of College Students

Exploring the Efficacy of Several Physiological Synchrony Methods During Collaborative Recall of Stories

Wearable Neurophysiological Recordings in Middle-School Classroom Correlate with Students’ Academic Performance

Understanding Physiological Responses of Students Over Different Courses

Detection of Dynamic Changes of Electrodermal Activity to Predict the Classroom Performance of College Students

Brain-to-Brain Synchrony and Learning Outcomes Vary by Student–Teacher Dynamics: Evidence from a Real-world Classroom Electroencephalography Study

Interpersonal Physiological Synchrony Predicts Group Cohesion

An Automatic Cardiac Arrhythmia Classification System with Wearable Electrocardiogram

Inter-brain Coupling Reflects Disciplinary Differences in Real-World Classroom Learning

Physiological Characterization of Student Engagement in the Naturalistic Classroom: A Mixed‐Methods Approach

Individual and Group-wise Classroom Seating Experience: Effects on Student Engagement in Different Courses

Behavioral and brain synchronization differences between expert and novice teachers when collaborating with students

Exploring Interactions and Regulations in Collaborative Learning: An Interdisciplinary Multimodal Dataset

Synchrony Expression Analysis of Human-Human Interaction Using Wearable Sensors.

Learning Attention Level Prediction Via Multimodal Physiological Data Using Wearable Wrist Devices

Learner's cognitive state recognition based on multimodal physiological signal fusion

Evaluating Children’s Engagement Through Wrist-worn Biosensors and LSTM Network

An Integrated Observing Technic for Collaborative Learning: The Multimodal Learning Analytics Based on the Video Coding and EEG Data Mining

Wearable Device-Based Real-Time Monitoring of Physiological Signals: Evaluating Cognitive Load Across Different Tasks