Abstract:Motion forecasting is crucial in enabling autonomous vehicles to anticipate the future trajectories of surrounding agents. To do so, it requires solving mapping, detection, tracking, and then forecasting problems, in a multi-step pipeline. In this complex system, advances in conventional forecasting methods have been made using curated data, i.e., with the assumption of perfect maps, detection, and tracking. This paradigm, however, ignores any errors from upstream modules. Meanwhile, an emerging end-to-end paradigm, that tightly integrates the perception and forecasting architectures into joint training, promises to solve this issue. However, the evaluation protocols between the two methods were so far incompatible and their comparison was not possible. In fact, conventional forecasting methods are usually not trained nor tested in real-world pipelines (e.g., with upstream detection, tracking, and mapping modules). In this work, we aim to bring forecasting models closer to the real-world deployment. First, we propose a unified evaluation pipeline for forecasting methods with real-world perception inputs, allowing us to compare conventional and end-to-end methods for the first time. Second, our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data. In particular, we show that this gap (1) stems not only from differences in precision but also from the nature of imperfect inputs provided by perception modules, and that (2) is not trivially reduced by simply finetuning on perception outputs. Based on extensive experiments, we provide recommendations for critical areas that require improvement and guidance towards more robust motion forecasting in the real world. The evaluation library for benchmarking models under standardized and practical conditions is provided: \url{<a class="link-external link-https" href="https://github.com/valeoai/MFEval" rel="external noopener nofollow">this https URL</a>}.

Injecting Planning-Awareness into Prediction and Detection Evaluation

Predictability Awareness for Efficient and Robust Multi-Agent Coordination

Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Autonomous Robots

Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles

A Control-Centric Benchmark for Video Prediction

Control-Aware Prediction Objectives for Autonomous Driving

Dynamically Switching Human Prediction Models for Efficient Planning

RAP: Risk-Aware Prediction for Robust Planning

How Does Perception Affect Safety: New Metrics and Strategy

Efficient Game-Theoretic Planning with Prediction Heuristic for Socially-Compliant Autonomous Driving

IR-STP: Enhancing Autonomous Driving With Interaction Reasoning in Spatio-Temporal Planning

Hybrid-Prediction Integrated Planning for Autonomous Driving

System Safety Monitoring of Learned Components Using Temporal Metric Forecasting

Planning by Simulation: Motion Planning with Learning-based Parallel Scenario Prediction for Autonomous Driving

Prediction Failure Risk-Aware Decision-Making for Autonomous Vehicles on Signalized Intersections

Evaluation Metrics for Object Detection for Autonomous Systems

A Predictive Autonomous Decision Aid for Calibrating Human-Autonomy Reliance in Multi-Agent Task Assignment

Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving

A Game-Theoretic Framework for Joint Forecasting and Planning

Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?