Abstract:Examination of dysfunctional brain dynamics often needs to tackle the difficulties of analyzing neural data of multiple modalities as recordings from a single source (such as EEG) do not always suffice in characterizing the brain states of interest. A typical example is sleep scoring from Polysomnography (PSG) for evaluation of sleep disorder, which has long been an onerous task and constantly results in unstable performance. Despite numerous successes achieved by the deep learning technologies recently booming along this direction, a grand challenge still remains: how to enable a stable solution to characterize the sleep stages (brain states) with temporal correlations embedded in the multi-modal recordings mainly from the brain activities without the support of sufficient a priori knowledge of the subjects.Aiming at the challenge, this study develops a deep learning framework for multi-modal sleep scoring. The framework uses a “dual-CNN” to simultaneously process inputs of two forms, i.e., temporal or time–frequency. The inputs of each branch may either consist of organized epochs of multiple modalities or their time–frequency representations. The fused deep features of the original multi-modal dataset will then be derived. The correlation of the original consecutive epochs embedded in the deep features can be reinforced through an RNN layer to aid classification with the final results fine-tuned by a customized Markov chain model, which considers the temporal correlations embedded in the deep features to amortize the potential bias resulted from the imbalance of brain states represented by the features.A case study of sleep scoring has been performed against a public Sleep-EDF dataset. Experimental results indicate that 1) the multi-modal features generated by the dual-CNN model help in significantly improve the accuracy of neural data classification; 2) the overall framework outperforms the state-of-the-art counterparts with an accuracy of 84.9% achieved in sleep scoring.

Cross-subject Drowsiness Recognition Model Based on the Multi-scale Parallel Convolution of EEG Channels

Subject-Independent Drowsiness Recognition from Single-Channel EEG with an Interpretable CNN-LSTM Model

A Compact and Interpretable Convolutional Neural Network for Cross-Subject Driver Drowsiness Detection from Single-Channel EEG

Benchmarking EEG-based Cross-dataset Driver Drowsiness Recognition with Deep Transfer Learning

Multi-source signal alignment and efficient multi-dimensional feature classification in the application of EEG-based subject-independent drowsiness detection

Drowsiness Detection Using Joint EEG-ECG Data With Deep Learning

Dual-CNN Based Multi-Modal Sleep Scoring with Temporal Correlation Driven Fine-Tuning

EEG_DMNet: A Deep Multi-Scale Convolutional Neural Network for Electroencephalography-Based Driver Drowsiness Detection

An EEG Channel Selection Framework for Driver Drowsiness Detection via Interpretability Guidance

EEG-Based Driver Drowsiness Estimation Using Convolutional Neural Networks

Dynamic threshold distribution domain adaptation network: a cross-subject fatigue recognition method based on EEG signals

Convolutional Neural Network for Drowsiness Detection Using EEG Signals

Cross-Subject Eeg-Based Emotion Recognition With Deep Domain Confusion

Single-channel EEG sleep staging based on data augmentation and cross-subject discrepancy alleviation

Multi-source domain adaptation based tempo-spatial convolution network for cross-subject EEG classification in RSVP task

Attention-Guided Multi-scale Convolutional Neural Network for Driving Fatigue Detection

Recognizing drowsiness in young men during real driving based on electroencephalography using an end-to-end deep learning approach

A subject-transfer framework for obviating inter- and intra-subject variability in EEG-based drowsiness detection

3DSleepNet: A Multi-Channel Bio-Signal Based Sleep Stages Classification Method Using Deep Learning

Customized deep learning algorithm for drowsiness detection using single-channel EEG signal

PSSS-EEG: A Probabilistic-masking Self-Supervised Swin-transformer model for EEG-based drowsiness recognition