Abstract:EEG-based unimodal method has demonstrated substantial success in the detection of driving fatigue. Nonetheless, the data from a single modality might be not sufficient to optimize fatigue detection due to incomplete information. To address this limitation and enhance the performance of driving fatigue detection, a novel multimodal architecture combining electroencephalography (EEG) and eye tracking data was proposed in this study. Specifically, EEG and eye tracking data were separately input into encoders, generating two one-dimensional (1D) features. Subsequently, these 1D features were fed into a cross-modal predictive alignment module to improve fusion efficiency and two 1D attention modules to enhance feature representation. Furthermore, the fused features were recognized by a linear classifier. To evaluate the effectiveness of the proposed multimodal method, comprehensive validation tasks were conducted, including intra-session, cross-session, and cross-subject evaluations. In the intra-session task, the proposed architecture achieves an exceptional average accuracy of 99.93%. Moreover, in the cross-session task, our method results in an average accuracy of 88.67%, surpassing the performance of EEG-only approach by 8.52%, eye tracking-only method by 5.92%, multimodal deep canonical correlation analysis (DCCA) technique by 0.42%, and multimodal deep generalized canonical correlation analysis (DGCCA) approach by 0.84%. Similarly, in the cross-subject task, the proposed approach achieves an average accuracy of 78.19%, outperforming EEG-only method by 5.87%, eye tracking-only approach by 4.21%, DCCA method by 0.55%, and DGCCA approach by 0.44%. The experimental results conclusively illustrate the superior effectiveness of the proposed method compared to both single modality approaches and canonical correlation analysis-based multimodal methods. Overall, this study provides a new and effective strategy for driving fatigue detection.

CEEMDAN Fuzzy Entropy Based Fatigue Driving Detection Using Single-Channel EEG

Driver Fatigue Detection Through Multiple Entropy Fusion Analysis in an EEG-based System

Constructing Multi-scale Entropy Based on the Empirical Mode Decomposition(EMD) and its Application in Recognizing Driving Fatigue

Eeg-Based Multiple Entropy Analysis For Assessing Driver Fatigue

Driving Fatigue Detection Based on Hybrid Electroencephalography and Eye Tracking

Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model

Driver fatigue state detection based on EEG and EOG feature fusion and deep learning

Fatigue driving detection method based on Time-Space-Frequency features of multimodal signals

Attention-Guided Multi-scale Convolutional Neural Network for Driving Fatigue Detection

A new feature selection approach for driving fatigue EEG detection with a modified machine learning algorithm

A Multi-Model Analysis for Driving Fatigue Detection using EEG Signals

Sample-Entropy-Based Method for Real Driving Fatigue Detection with Multichannel Electroencephalogram

Multiple nonlinear features fusion based driving fatigue detection

Research on Fatigue Driving Detection Using Forehead EEG Based on Adaptive Multi-Scale Entropy

Detecting driving fatigue with multimodal deep learning

A Novel Fatigue Driving State Recognition and Warning Method Based on EEG and EOG Signals

Ensemble Classifier for Driver's Fatigue Detection Based on a Single EEG Channel

Classification of EEG Signals from Driving Fatigue by Image-Based Deep Recurrent Neural Networks

Fatigue Driving State Detection Based on Spatial Characteristics of EEG Signals

EEG and ECG-Based Multi-Sensor Fusion Computing for Real-Time Fatigue Driving Recognition Based on Feedback Mechanism

Dynamic threshold distribution domain adaptation network: a cross-subject fatigue recognition method based on EEG signals