Abstract:Scene understanding, defined as learning, extraction, and representation of interactions among traffic elements, is one of the critical challenges toward high-level autonomous driving (AD). Current scene understanding methods mainly focus on one concrete single task, such as trajectory prediction and risk level evaluation. Although they perform well on specific metrics, the generalization ability is insufficient to adapt to the real traffic complexity and downstream demand diversity. In this study, we propose PreGSU, a generalized pre-trained scene understanding model based on graph attention network to learn the universal interaction and reasoning of traffic scenes to support various downstream tasks. After the feature engineering and sub-graph module, all elements are embedded as nodes to form a dynamic weighted graph. Then, four graph attention layers are applied to learn the relationships among agents and lanes. In the pre-train phase, the understanding model is trained on two self-supervised tasks: Virtual Interaction Force (VIF) modeling and Masked Road Modeling (MRM). Based on the artificial potential field theory, VIF modeling enables PreGSU to capture the agent-to-agent interactions while MRM extracts agent-to-road connections. In the fine-tuning process, the pre-trained parameters are loaded to derive detailed understanding outputs. We conduct validation experiments on two downstream tasks, i.e., trajectory prediction in urban scenario, and intention recognition in highway scenario, to verify the generalized ability and understanding ability. Results show that compared with the baselines, PreGSU achieves better accuracy on both tasks, indicating the potential to be generalized to various scenes and targets. Ablation study shows the effectiveness of pre-train task design.

Self Supervised Clustering of Traffic Scenes using Graph Representations

SCENE: Reasoning About Traffic Scenes Using Heterogeneous Graph Neural Networks

Graph Convolutional Networks for Complex Traffic Scenario Classification

Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphs

A Bottom-up Paradigm for Traffic Scene Graph Representation

Identifying Traffic Clusters in Urban Networks Based on Graph Theory Using License Plate Recognition Data

A novel unsupervised approach to discovering regions of interest in traffic images

SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphs

Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding

Fingerprint of a Traffic Scene: an Approach for a Generic and Independent Scene Assessment

Learning Semantic Scene Models by Object Classification and Trajectory Clustering

nuScenes Knowledge Graph -- A comprehensive semantic representation of traffic scenes for trajectory prediction

Road Scene Graph: A Semantic Graph-Based Scene Representation Dataset for Intelligent Vehicles

Toward Driving Scene Understanding: A Paradigm and Benchmark Dataset for Ego-Centric Traffic Scene Graph Representation

Spatiotemporal Analysis of Static and Dynamic Traffic Elements from Road Scenes.

Sparse-Representation-Based Graph Embedding for Traffic Sign Recognition

Vehicle-Related Scene Segmentation Using CapsNets.

Utilizing Hybrid Trajectory Prediction Models to Recognize Highly Interactive Traffic Scenarios

Graph-Based Approaches to Clustering Network-Constrained Trajectory Data

GeneAnnotator: A Semi-automatic Annotation Tool for Visual Scene Graph

PreGSU-A Generalized Traffic Scene Understanding Model for Autonomous Driving based on Pre-trained Graph Attention Network