Abstract:Predicting the future behavior of agents is a fundamental task in autonomous vehicle domains. Accurate prediction relies on comprehending the surrounding map, which significantly regularizes agent behaviors. However, existing methods have limitations in exploiting the map and exhibit a strong dependence on historical trajectories, which yield unsatisfactory prediction performance and robustness. Additionally, their heavy network architectures impede real-time applications. To tackle these problems, we propose Map-Agent Coupled Transformer (MacFormer) for real-time and robust trajectory prediction. Our framework explicitly incorporates map constraints into the network via two carefully designed modules named coupled map and reference extractor. A novel multi-task optimization strategy (MTOS) is presented to enhance learning of topology and rule constraints. We also devise bilateral query scheme in context fusion for a more efficient and lightweight network. We evaluated our approach on Argoverse 1, Argoverse 2, and nuScenes real-world benchmarks, where it all achieved state-of-the-art performance with the lowest inference latency and smallest model size. Experiments also demonstrate that our framework is resilient to imperfect tracklet inputs. Furthermore, we show that by combining with our proposed strategies, classical models outperform their baselines, further validating the versatility of our framework.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to achieve real - time and robust trajectory prediction in the field of autonomous vehicles. Specifically, the paper points out that existing methods have limitations in using map information and mainly rely on historical trajectories, resulting in poor prediction performance and robustness. In addition, the network architectures of these methods are heavy and it is difficult to achieve real - time applications. To overcome these problems, the paper proposes a new framework named Map - Agent Coupled Transformer (MacFormer), aiming to improve the accuracy and robustness of trajectory prediction by directly coupling map constraints with agent motion while maintaining lightweight and efficient characteristics. ### Main Contributions: 1. **Map - Agent Coupled Framework**: A new map - agent coupled framework is proposed, which can effectively integrate map constraints into the system, achieving this goal through coupling the map and reference extractors and the multi - task optimization strategy (MTOS). 2. **Efficient Context Fusion Scheme**: A bilateral query scheme is designed, allowing parallel context fusion between the map and the agent, thus significantly reducing the time and space complexity. 3. **State - of - the - Art Performance**: Evaluations are carried out on multiple large - scale real - world benchmark datasets (such as Argoverse 1&2 and nuScenes), and the results show that MacFormer not only achieves state - of - the - art performance but also has lower inference latency and fewer parameters. ### Method Overview: - **Coupled Layer**: Extract historical motion features and construct a coupled map, and calculate the historical relative motion of the map and the agent at each timestamp. - **Reference Extractor**: Learn map - related references from the coupled map features to guide the spatial distribution of the predicted trajectory. - **Multi - Task Optimization Strategy (MTOS)**: Ensure that the prediction results are affected by map constraints through the multi - task optimization strategy, including the coupled motion task, the motion capture task, and the main prediction task. - **Bilateral Query**: Promote effective context fusion between the map and the agent through the bilateral query strategy to improve computational efficiency. ### Experimental Results: - **Performance Improvement**: On multiple benchmark datasets, MacFormer achieves the lowest inference latency and the smallest model size while reaching state - of - the - art performance. - **Robustness**: The experiments also show that this framework has strong robustness against imperfect trajectory inputs. - **Applicability**: Combined with the proposed strategies, the performance of the classical model is also improved, further verifying the universality of the framework. In conclusion, through proposing the MacFormer framework, this paper solves the deficiencies of existing trajectory prediction methods in using map information and real - time performance, providing a new solution for trajectory prediction in the field of autonomous driving.

MacFormer: Map-Agent Coupled Transformer for Real-time and Robust Trajectory Prediction

MacFormer: Map-Agent Coupled Transformer for Real-Time and Robust Trajectory Prediction

Dynamic-learning Spatial-Temporal Transformer Network for Vehicular Trajectory Prediction at Urban Intersections

Multi‐agent trajectory prediction with adaptive perception‐guided transformers

LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Prediction

Spatio-Temporal Context Graph Transformer Design for Map-Free Multi-Agent Trajectory Prediction

Map-Free Trajectory Prediction with Map Distillation and Hierarchical Encoding

Edge-Enriched Graph Transformer for Multiagent Trajectory Prediction With Relative Positional Semantics

Multi-scale Temporal Fusion Transformer for Incomplete Vehicle Trajectory Prediction

MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying

Multi‐future Transformer: Learning Diverse Interaction Modes for Behaviour Prediction in Autonomous Driving

Context‐aware trajectory prediction for autonomous driving in heterogeneous environments

Single-Agent LSTM Encoders Single-Agent LSTM Decoders Output : Multi-Agent Future Trajectories Input 1 : Multi-Agent Past Trajectories Input 2 : Scene Context Image Scene Context Channels Multi-Agent Channels Convolutional Operator : Multi-Agent Tensor Fusion

QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Prediction

Crossmodal Transformer Based Generative Framework for Pedestrian Trajectory Prediction

Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding

Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

Lane Transformer: A High-Efficiency Trajectory Prediction Model

Agents-Separated Prediction-Former: A Novel Multi-Agent Trajectory Prediction Model Based on Transformer with 2D Input

ControlMTR: Control-Guided Motion Transformer with Scene-Compliant Intention Points for Feasible Motion Prediction

TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses