Abstract:Accurate behavior prediction for vehicles is essential but challenging for autonomous driving. Most existing studies show satisfying performance under regular scenarios, but most neglected safety-critical scenarios. In this study, a spatio-temporal dual-encoder network named STDA for safety-critical scenarios was developed. Considering the exceptional capabilities of human drivers in terms of situational awareness and comprehending risks, driver attention was incorporated into STDA to facilitate swift identification of the critical regions, which is expected to improve both performance and interpretability. STDA contains four parts: the driver attention prediction module, which predicts driver attention; the fusion module designed to fuse the features between driver attention and raw images; the temporary encoder module used to enhance the capability to interpret dynamic scenes; and the behavior prediction module to predict the behavior. The experiment data are used to train and validate the model. The results show that STDA improves the G-mean from 0.659 to 0.719 when incorporating driver attention and adopting a temporal encoder module. In addition, extensive experimentation has been conducted to validate that the proposed module exhibits robust generalization capabilities and can be seamlessly integrated into other mainstream models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in autonomous and semi - autonomous vehicles, how to accurately predict driver behavior, especially in safety - critical scenarios. Most of the existing research performs well in normal scenarios, but has poor performance in safety - critical scenarios. Therefore, this paper proposes a new spatio - temporal dual - encoder network (STDA), which improves the prediction accuracy of driver behavior by introducing driver attention. ### Specific problems and solutions: 1. **Problem description**: - **Limitations of existing models**: Traditional models mainly rely on external environmental factors and historical driving data, ignoring the important factor of how drivers focus their attention in different situations. - **Importance of safety - critical scenarios**: In these scenarios, timely and accurate prediction of driver behavior is crucial for ensuring traffic safety. 2. **Solutions**: - **Introducing driver attention**: By simulating the situational awareness and risk - understanding ability of human drivers, driver attention is incorporated into the model to quickly identify key areas, thereby improving prediction performance and the interpretability of the model. - **Designing the STDA model**: The STDA model contains four main modules: 1. **Driver attention prediction module**: Predicts the attention distribution of the driver. 2. **Fusion module**: Fuses driver attention with the original image features. 3. **Temporal encoding module**: Enhances the model's understanding ability of dynamic scenes. 4. **Behavior prediction module**: Finally predicts the driver's behavior. ### Model improvement effects: - **Experimental results**: Through experimental verification, after introducing driver attention and using the temporal encoding module, the G - mean of the STDA model has increased from 0.659 to 0.719, indicating that it has significant advantages in dealing with unbalanced data sets and safety - critical scenarios. - **Robustness and universality**: Extensive experiments further prove that the proposed modules have strong generalization ability and the ability to be seamlessly integrated with other mainstream models. In summary, this paper aims to improve the accuracy of driver behavior prediction in safety - critical scenarios by introducing driver attention and improving the model architecture, thereby providing more reliable support for autonomous driving technology.

STDA: Spatio-Temporal Dual-Encoder Network Incorporating Driver Attention to Predict Driver Behaviors Under Safety-Critical Scenarios

A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidents

When Do Drivers Concentrate? Attention-based Driver Behavior Modeling With Deep Reinforcement Learning

Multisource Adaption for Driver Attention Prediction in Arbitrary Driving Scenes

Gated Driver Attention Predictor

Improved Dynamic Spatial-Temporal Attention Network for Early Anticipation of Traffic Accidents

FDAN: Fuzzy deep attention networks for driver behavior recognition

Temporal Information Fusion Network for Driving Behavior Prediction.

E-DNet: An End-to-End Dual-Branch Network for Driver Steering Intention Detection

An ADAS with better driver satisfaction under rear-end near-crash scenarios: A spatio-temporal graph transformer-based prediction framework of evasive behavior and collision risk

A Multimodal Data-Driven Approach for Driving Risk Assessment

DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by A Benchmark

Lane Changing Maneuver Prediction by Using Driver’s Spatio-Temporal Gaze Attention Inputs for Naturalistic Driving

M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving

Improving real-time driver distraction detection via constrained attention mechanism

AHMF: Adaptive Hybrid-Memory-Fusion Model for Driver Attention Prediction

DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification

Context-Aware Attention Encoder-Decoder Network for Connected Heavy-Duty Vehicle Aggressive Driving Identification under Naturalistic Driving Conditions

Driver attention prediction based on convolution and transformers

DSTANet: learning a dual-stream model for anomaly driving action detection using spatio-temporal and appearance features

Enhancing autonomous driving systems with deep learning and spatial channel attention mechanisms: an experimental study