Abstract:With the unprecedented shift towards automated urban environments in recent years, a new paradigm is required to study pedestrian behaviour. Studying pedestrian behaviour in futuristic scenarios requires modern data sources that consider both the Automated Vehicle (AV) and pedestrian perspectives. Current open datasets on AVs predominantly fail to account for the latter, as they do not include an adequate number of events and associated details that involve pedestrian and vehicle interactions. To address this issue, we propose using Virtual Reality (VR) data as a complementary resource to current datasets, which can be designed to measure pedestrian behaviour under specific conditions. In this research, we focus on the context-aware pedestrian trajectory prediction framework for automated vehicles at mid-block unsignalized crossings. For this purpose, we develop a novel multi-input network of Long Short-Term Memory (LSTM) and fully connected dense layers. In addition to past trajectories, the proposed framework incorporates pedestrian head orientations and distance to the upcoming vehicles as sequential input data. By merging the sequential data with contextual information of the environment, we train a model to predict the future pedestrian trajectory. Our results show that the prediction error and overfitting to the training data are reduced by considering contextual information in the model. To analyze the application of the methods to real AV data, the proposed framework is trained and applied to pedestrian trajectory extracted from an open-access video dataset. Finally, by implementing a game theory-based model interpretability method, we provide detailed insights and propose recommendations to improve the current automated vehicle sensing systems from a pedestrian-oriented point of view.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to predict the trajectories of pedestrians crossing the road at sections without traffic lights in the environment of autonomous vehicles (AVs). Specifically, the paper focuses on the following aspects: 1. **Limitations of existing data sets**: Most of the currently publicly available autonomous vehicle data sets lack sufficient pedestrian - vehicle interaction events, especially data on pedestrians crossing sections without traffic lights. These data sets fail to provide enough details to study pedestrian behavior, especially in the autonomous driving environment. 2. **Importance of pedestrian behavior prediction**: In the absence of a driver, autonomous vehicles need to be able to predict the intentions, choices, and movement trajectories of pedestrians in order to avoid traffic accidents. In recent years, accidents due to the failure to accurately predict pedestrian behavior have occurred, highlighting the urgency of studying this problem. 3. **Introduction of virtual reality (VR) data**: To make up for the deficiencies of existing data sets, the paper proposes to use virtual reality data as a supplementary resource and measure pedestrian behavior by designing experiments under specific conditions. The virtual reality environment can simulate complex traffic scenes and record pedestrians' reactions, including their positions, head orientations, and distances from approaching vehicles. 4. **Development of a new prediction framework**: The paper develops a multi - input network, combining long - short - term memory (LSTM) and fully - connected layers, to predict the future trajectories of pedestrians. This framework not only considers past trajectories but also combines sequential data such as pedestrian head orientations and distances from approaching vehicles, as well as environmental context information. 5. **Model interpretability**: To analyze the prediction accuracy of the model, the paper applies a posteriori interpretation method based on game theory, provides detailed insights, and makes improvement suggestions to enhance the performance of the current autonomous vehicle perception system from the perspective of pedestrians. In summary, the main objective of this paper is to improve the ability of autonomous vehicles to predict pedestrian trajectories at sections without traffic lights by introducing virtual reality data and developing a new prediction framework, thereby enhancing road safety.

A context-aware pedestrian trajectory prediction framework for automated vehicles

Decoding pedestrian and automated vehicle interactions using immersive virtual reality and interpretable deep learning

Pedestrian Trajectory Prediction in Pedestrian-Vehicle Mixed Environments: A Systematic Review

Pedestrian Trajectory Prediction for Real-Time Autonomous Systems via Context-Augmented Transformer Networks

Analysis and Prediction of Pedestrian Crosswalk Behavior during Automated Vehicle Interactions

Enhanced Multimodal Trajectory Prediction for Autonomous Vehicles Using Advanced Diffusion Model Techniques

Crossmodal Transformer Based Generative Framework for Pedestrian Trajectory Prediction

Pedestrian Trajectory Prediction using Context-Augmented Transformer Networks

Deep learning-based pedestrian trajectory prediction and risk assessment at signalized intersections using trajectory data captured through roadside LiDAR

A Virtual Reality Framework for Human-Driver Interaction Research: Safe and Cost-Effective Data Collection

Off The Beaten Sidewalk: Pedestrian Prediction In Shared Spaces For Autonomous Vehicles

Pedestrian Trajectory Prediction Combining Probabilistic Reasoning and Sequence Learning

A Survey on Motion Prediction of Pedestrians and Vehicles for Autonomous Driving

Learning Pedestrian Actions to Ensure Safe Autonomous Driving

Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction

Understanding Pedestrian-Vehicle Interactions with Vehicle Mounted Vision: An LSTM Model and Empirical Analysis

Pedestrian Trajectory Prediction at Un-Signalized Intersection Using Probabilistic Reasoning and Sequence Learning

Context Model for Pedestrian Intention Prediction using Factored Latent-Dynamic Conditional Random Fields

A Multimodal Trajectory Prediction Method for Pedestrian Crossing Considering Pedestrian Motion State

A Review of Trajectory Prediction Methods for the Vulnerable Road User

Pedestrian Crossing Intention Forecasting at Unsignalized Intersections Using Naturalistic Trajectories