Abstract:EEG-based unimodal method has demonstrated substantial success in the detection of driving fatigue. Nonetheless, the data from a single modality might be not sufficient to optimize fatigue detection due to incomplete information. To address this limitation and enhance the performance of driving fatigue detection, a novel multimodal architecture combining electroencephalography (EEG) and eye tracking data was proposed in this study. Specifically, EEG and eye tracking data were separately input into encoders, generating two one-dimensional (1D) features. Subsequently, these 1D features were fed into a cross-modal predictive alignment module to improve fusion efficiency and two 1D attention modules to enhance feature representation. Furthermore, the fused features were recognized by a linear classifier. To evaluate the effectiveness of the proposed multimodal method, comprehensive validation tasks were conducted, including intra-session, cross-session, and cross-subject evaluations. In the intra-session task, the proposed architecture achieves an exceptional average accuracy of 99.93%. Moreover, in the cross-session task, our method results in an average accuracy of 88.67%, surpassing the performance of EEG-only approach by 8.52%, eye tracking-only method by 5.92%, multimodal deep canonical correlation analysis (DCCA) technique by 0.42%, and multimodal deep generalized canonical correlation analysis (DGCCA) approach by 0.84%. Similarly, in the cross-subject task, the proposed approach achieves an average accuracy of 78.19%, outperforming EEG-only method by 5.87%, eye tracking-only approach by 4.21%, DCCA method by 0.55%, and DGCCA approach by 0.44%. The experimental results conclusively illustrate the superior effectiveness of the proposed method compared to both single modality approaches and canonical correlation analysis-based multimodal methods. Overall, this study provides a new and effective strategy for driving fatigue detection.

Constructing Multi-scale Entropy Based on the Empirical Mode Decomposition(EMD) and its Application in Recognizing Driving Fatigue

Eeg-Based Multiple Entropy Analysis For Assessing Driver Fatigue

Driver Fatigue Detection Through Multiple Entropy Fusion Analysis in an EEG-based System

Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model

A new feature selection approach for driving fatigue EEG detection with a modified machine learning algorithm

Research on Recognition Method of Driving Fatigue State Based on Sample Entropy and Kernel Principal Component Analysis

An Algorithm for Extracting Entropy Features from EEG Signals Based on T-test and KPCA and Its Application on Driving Fatigue State Recognition

Noise Robustness Analysis of Performance for Eeg-Based Driver Fatigue Detection Using Different Entropy Feature Sets

Multiple nonlinear features fusion based driving fatigue detection

Discrimination of Driver Fatigue Based on Distortion Energy Density Theory and Multiple Physiological Signals

Driving Fatigue Detection Based on Hybrid Electroencephalography and Eye Tracking

Fatigue driving detection method based on Time-Space-Frequency features of multimodal signals

A Multi-Model Analysis for Driving Fatigue Detection using EEG Signals

Driving Drowsiness Detection With Eeg Using A Modified Hierarchical Extreme Learning Machine Algorithm With Particle Swarm Optimization: A Pilot Study

Feature extraction of EEG signals based on functional data analysis and its application to recognition of driver fatigue state

A Multi-Feature Fusion and Situation Awareness-Based Method for Fatigue Driving Level Determination

Transferable Takagi-Sugeno-Kang Fuzzy Classifier With Multi-Views for EEG-Based Driving Fatigue Recognition in Intelligent Transportation

Attention-Guided Multi-scale Convolutional Neural Network for Driving Fatigue Detection

Driver fatigue detection method based on multi-feature empirical fusion model

Ensemble Classifier for Driver's Fatigue Detection Based on a Single EEG Channel

Fatigue Detection With Covariance Manifolds of Electroencephalography in Transportation Industry