Abstract:It is crucial to evaluate learning outcomes by identifying the cognitive state of the learner during the learning process. Studies utilizing Electroencephalography (EEG) and other peripheral physiological signals, combined with deep learning models, have demonstrated improved performance in cognitive state recognition. These studies have primarily focused on unimodal data, which are vulnerable to various types of noise, making it difficult to fully capture and represent cognitive states. Leveraging the complementarity between multimodal physiological signals can mitigate the impact of anomalies in unimodal data, thereby improving the accuracy and stability of cognitive state recognition. Therefore, this study proposes a multimodal physiological signal feature representation fusion model based on multi-level attention (PSFMMA). The model aims to integrate multimodal physiological signals to identify learners' cognitive states with greater stability and accuracy. PSFMMA first extracts the temporal features of physiological signals by multiplexing the embedding layer. Subsequently, it generates signal representation vectors by further extracting semantic features through a signal feature mapping layer and enhancing important features with designed attention modules. Finally, the model employs an attention mechanism based on different signal representation vectors to fuse multimodal information for identifying learners' cognitive states. This study designs various learning activities and collects electroencephalography (EEG), electrodermal activity (EDA), and photoplethysmography (PPG) data from 22 participants engaging in these activities to create the Based on Learning Activities Collection (BLAC) dataset. The proposed model was evaluated on the BLAC dataset, achieving an identification accuracy of 96.32 ± 0.32%. The results demonstrate that the model can effectively recognize learners' cognitive states. Furthermore, the model's performance was validated on the publicly available emotion classification dataset DEAP, attaining an accuracy of 99.15 ± 0.12%. The source code is available at https://github.com/chengshudaxuesheng/PSFMMA.

Learner's cognitive state recognition based on multimodal physiological signal fusion

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Emotion recognition based on multiple physiological signals

Emotion Recognition from Multiple Physiological Signals Using Intra- and Inter-Modality Attention Fusion Network

Learning Unit State Recognition Based on Multi-channel Data Fusion

Multimodal Latent Emotion Recognition from Micro-expression and Physiological Signals

Recognition of Emotions Using Multimodal Physiological Signals and an Ensemble Deep Learning Model

A Data-Driven Framework for Learners' Cognitive Load Detection Using ECG-PPG Physiological Feature Fusion and XGBoost Classification.

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Affective EEG-Based Person Identification With Continual Learning

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions.

A Fusion Framework for Confusion Analysis in Learning Based on EEG Signals

Cognitive Load Prediction from Multimodal Physiological Signals Using Multiview Learning.

Multimodal Emotion Recognition Model using Physiological Signals

CiABL: Completeness-induced Adaptative Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Learning Attention Level Prediction Via Multimodal Physiological Data Using Wearable Wrist Devices

Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model

Emotion Recognition Based on a Limited Number of Multimodal Physiological Signals Channels

Automatic Emotion Recognition Using Temporal Multimodal Deep Learning