Abstract:EEG-based unimodal method has demonstrated substantial success in the detection of driving fatigue. Nonetheless, the data from a single modality might be not sufficient to optimize fatigue detection due to incomplete information. To address this limitation and enhance the performance of driving fatigue detection, a novel multimodal architecture combining electroencephalography (EEG) and eye tracking data was proposed in this study. Specifically, EEG and eye tracking data were separately input into encoders, generating two one-dimensional (1D) features. Subsequently, these 1D features were fed into a cross-modal predictive alignment module to improve fusion efficiency and two 1D attention modules to enhance feature representation. Furthermore, the fused features were recognized by a linear classifier. To evaluate the effectiveness of the proposed multimodal method, comprehensive validation tasks were conducted, including intra-session, cross-session, and cross-subject evaluations. In the intra-session task, the proposed architecture achieves an exceptional average accuracy of 99.93%. Moreover, in the cross-session task, our method results in an average accuracy of 88.67%, surpassing the performance of EEG-only approach by 8.52%, eye tracking-only method by 5.92%, multimodal deep canonical correlation analysis (DCCA) technique by 0.42%, and multimodal deep generalized canonical correlation analysis (DGCCA) approach by 0.84%. Similarly, in the cross-subject task, the proposed approach achieves an average accuracy of 78.19%, outperforming EEG-only method by 5.87%, eye tracking-only approach by 4.21%, DCCA method by 0.55%, and DGCCA approach by 0.44%. The experimental results conclusively illustrate the superior effectiveness of the proposed method compared to both single modality approaches and canonical correlation analysis-based multimodal methods. Overall, this study provides a new and effective strategy for driving fatigue detection.

Eeg-Based Multiple Entropy Analysis For Assessing Driver Fatigue

Driver Fatigue Detection Through Multiple Entropy Fusion Analysis in an EEG-based System

Ensemble Classifier for Driver's Fatigue Detection Based on a Single EEG Channel

Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model

Noise Robustness Analysis of Performance for Eeg-Based Driver Fatigue Detection Using Different Entropy Feature Sets

Constructing Multi-scale Entropy Based on the Empirical Mode Decomposition(EMD) and its Application in Recognizing Driving Fatigue

Driver fatigue EEG signals detection by using robust univariate analysis

Automated detection of driver fatigue based on EEG signals using gradient boosting decision tree model

A Multi-Model Analysis for Driving Fatigue Detection using EEG Signals

Discrimination of Driver Fatigue Based on Distortion Energy Density Theory and Multiple Physiological Signals

Fatigue driving detection technology based on EEG signal

Multiple nonlinear features fusion based driving fatigue detection

A new feature selection approach for driving fatigue EEG detection with a modified machine learning algorithm

Driving Drowsiness Detection With Eeg Using A Modified Hierarchical Extreme Learning Machine Algorithm With Particle Swarm Optimization: A Pilot Study

Attention-Guided Multi-scale Convolutional Neural Network for Driving Fatigue Detection

Exploring the fatigue affecting electroencephalography based functional brain networks during real driving in young males

Research on Recognition Method of Driving Fatigue State Based on Sample Entropy and Kernel Principal Component Analysis

Fatigue Driving State Detection Based on Spatial Characteristics of EEG Signals

Driving Fatigue Detection Based on Hybrid Electroencephalography and Eye Tracking

An Algorithm for Extracting Entropy Features from EEG Signals Based on T-test and KPCA and Its Application on Driving Fatigue State Recognition

Compact Vehicle Driver Fatigue Recognition Technology Based on EEG Signal