Assessment of Situation Awareness in Automated Driving Using Eye-fixation Related EEG Analysis
Jing Yang,Nade Liang,K. Prakah-Asante,Reates Curry,Mike Blommer,Radhakrishnan Swaminathan,Brandon J. Pitts,Denny Yu
DOI: https://doi.org/10.1177/1071181322661233
2022-09-01
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
Abstract:A growing percentage of a new automated vehicles is classified Society of Automotive Engineers (SAE) Level 3, which can perform most driving tasks but may require occasional human intervention. To successfully take over control of the vehicle when needed, the driver must have good situation awareness (SA) (Endsley, 1995). Consequently, an accurate assessment of drivers' SA is crucial to evaluating one’s ability to manage takeover events. SA is often measured using questionnaires, e.g., the Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 2021). Direct measures of SA, however, can be intrusive and subjective, thus limiting their implementation in real-life situations. Utilizing physiological sensors, such as eye trackers, can strengthen SA assessments. Several studies have attempted to monitor drivers' SA using eye-tracking (included in Zhang et al., 2020 review), but these studies may have not considered the look-but-fail-tosee phenomenon (Robbins et al., 2019). As different stages of SA require various mental resources and cognitive processes, the incorporation of electroencephalography (EEG) measurements that capture those cognitive processes can support a deeper understanding of SA. This can be achieved using eye fixation-related EEG (FRE) analysis (Frey et al., 2018), which enables the analysis of neuropsychological processes while assessing the time when eye positions are focused on target objects. However, to the best of the authors' knowledge, no study has applied FRE to assess SA. Therefore, the objective of this study was to evaluate the feasibility of using FRE analysis to assess SA. Thirty participants were performed three SAE L3 automated driving tasks using the miniSim driving simulator. There were two phases during each driving task: a pre-takeover (pre-TOR) and a post-takeover (post-TOR). The pre-TOR phase lasted seven minutes, during which participants engaged in three different types of non-driving related (NDRTs) that varied in terms of required visual attention allocation. In the post-TOR segment, drivers were expected to end engagement in the NDRT and gather information from the driving environment in preparation for a vehicle-to-driver transition. Participants’ EEG and eye-tracker signals were collected throughout the experiment using EPOC+ (Emotiv, 14 channels) and FOVIO FX3 systems, respectively. The SAGAT was used as the direct SA measure after each driving task. The analysis first defined the areas of interest (AOIs) as the areas surrounding the objects examined in the SAGAT questions. We also analyzed raw gaze points to identify fixations (i.e., duration > 75ms) in AOIs. The EEG signals, time-locked to the onsets of the AOI-fixations (i.e., FRE), were then extracted. FREs with durations of 300ms, 400ms, 500ms, and 600ms were extracted and compared to determine the best observation window for SA. Moreover, the spectral densities of theta (4 8 Hz), alpha (8 13 Hz), and beta (13 30 Hz) bands of each EEG channel were calculated for each extracted FRE. Based on the SAGAT scores related to particular AOIs, the ground truth of each feature was derived. For each FRE, one of three possible SA labels could be assigned: 1) no SA, neither perception nor comprehension SAGAT questions were correct, 2) perception SA (only perception SAGAT questions were correct), and 3) comprehension SA (comprehension SAGAT questions were correct). Five classification models were considered for the SA level classification: k-Nearest Neighbor, Naive Bayes, tree boosting classifier XGboost, Support Vector Machine, and Neural Network Model. The results indicated that the FRE observation window with a duration of 500ms achieved the best classification performance with an accuracy of 75.1% using the neural network model. This was in line with our expectations, as EEG activities 500ms after a stimulus has been reported to be an essential event-related potential, which can be elicited by external visual, auditory, and somatosensory stimuli (Zhong et al., 2019). Given that a higher level of SA is crucial for driving, integrating an EEG and eye-tracker through the FRE method can prove especially beneficial in understanding if perceived information was stored in the working memory of the driver for future comprehension and projection. Ultimately, this work can help inform the design of in-vehicle monitoring systems for SAE L3 vehicle operations.