Abstract:EEG-based unimodal method has demonstrated substantial success in the detection of driving fatigue. Nonetheless, the data from a single modality might be not sufficient to optimize fatigue detection due to incomplete information. To address this limitation and enhance the performance of driving fatigue detection, a novel multimodal architecture combining electroencephalography (EEG) and eye tracking data was proposed in this study. Specifically, EEG and eye tracking data were separately input into encoders, generating two one-dimensional (1D) features. Subsequently, these 1D features were fed into a cross-modal predictive alignment module to improve fusion efficiency and two 1D attention modules to enhance feature representation. Furthermore, the fused features were recognized by a linear classifier. To evaluate the effectiveness of the proposed multimodal method, comprehensive validation tasks were conducted, including intra-session, cross-session, and cross-subject evaluations. In the intra-session task, the proposed architecture achieves an exceptional average accuracy of 99.93%. Moreover, in the cross-session task, our method results in an average accuracy of 88.67%, surpassing the performance of EEG-only approach by 8.52%, eye tracking-only method by 5.92%, multimodal deep canonical correlation analysis (DCCA) technique by 0.42%, and multimodal deep generalized canonical correlation analysis (DGCCA) approach by 0.84%. Similarly, in the cross-subject task, the proposed approach achieves an average accuracy of 78.19%, outperforming EEG-only method by 5.87%, eye tracking-only approach by 4.21%, DCCA method by 0.55%, and DGCCA approach by 0.44%. The experimental results conclusively illustrate the superior effectiveness of the proposed method compared to both single modality approaches and canonical correlation analysis-based multimodal methods. Overall, this study provides a new and effective strategy for driving fatigue detection.

Analyzing pilots’ fatigue for prolonged flight missions: Multimodal analysis approach using vigilance test and eye tracking

Pilot Fatigue Evaluation Based on Eye-Movement Index and Performance Index.

Assessment Of Pilots Mental Fatigue Status With The Eye Movement Features

Pilot's visual attention allocation modeling under fatigue

Enhancing pilot vigilance assessment: The role of flight data and continuous performance test in detecting random attention loss in short IFR flights

Evaluation of Fatigue Induced During the Flight Tasks

A Sensitivity Analysis Method Of Physiological Signals For Assessing Pilot Workload

Detection of mental fatigue state using heart rate variability and eye metrics during simulated flight

Real-time Evaluation Method of Flight Mission Load Based on Sensitivity Analysis of Physiological Factors

Evaluation of eye metrics as a detector of fatigue

Proposed Data-Driven Approach for Occupational Risk Management of Aircrew Fatigue

A comprehensive prediction and evaluation method of pilot workload

Assessment of early onset of driver fatigue using multimodal fatigue measures in a static simulator

Fatigue Detection of Air Traffic Controllers Through Their Eye Movements

Driving Fatigue Detection Based on Hybrid Electroencephalography and Eye Tracking

Correlation Between Physiological and Performance-Based Metrics to Estimate Pilots' Cognitive Workload

Assessment of Ocular and Physiological Metrics to Discriminate Flight Phases in Real Light Aircraft

Use of multimodal physiological signals to explore pilots’ cognitive behaviour during flight strike task performance

Multiple Physiological Indexes Analysis in Commercial Flight Tasks for Fatigue Evaluation

Effects of sleep/wake history and circadian phase on proposed pilot fatigue safety performance indicators

Modeling and analysis of fatigue detection with multi-channel data fusion