Abstract:Motion forecasting is crucial in enabling autonomous vehicles to anticipate the future trajectories of surrounding agents. To do so, it requires solving mapping, detection, tracking, and then forecasting problems, in a multi-step pipeline. In this complex system, advances in conventional forecasting methods have been made using curated data, i.e., with the assumption of perfect maps, detection, and tracking. This paradigm, however, ignores any errors from upstream modules. Meanwhile, an emerging end-to-end paradigm, that tightly integrates the perception and forecasting architectures into joint training, promises to solve this issue. However, the evaluation protocols between the two methods were so far incompatible and their comparison was not possible. In fact, conventional forecasting methods are usually not trained nor tested in real-world pipelines (e.g., with upstream detection, tracking, and mapping modules). In this work, we aim to bring forecasting models closer to the real-world deployment. First, we propose a unified evaluation pipeline for forecasting methods with real-world perception inputs, allowing us to compare conventional and end-to-end methods for the first time. Second, our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data. In particular, we show that this gap (1) stems not only from differences in precision but also from the nature of imperfect inputs provided by perception modules, and that (2) is not trivially reduced by simply finetuning on perception outputs. Based on extensive experiments, we provide recommendations for critical areas that require improvement and guidance towards more robust motion forecasting in the real world. The evaluation library for benchmarking models under standardized and practical conditions is provided: \url{<a class="link-external link-https" href="https://github.com/valeoai/MFEval" rel="external noopener nofollow">this https URL</a>}.

The Future of Forecasting Competitions: Design Attributes and Principles

Forecasting in social settings: The state of the art

Forecasting: theory and practice

Designing forecasting software for forecast users: Empowering non-experts to create and understand their own forecasts

Forecasting Competitions with Correlated Events

Asking Better Questions -- The Art and Science of Forecasting: A mechanism for truer answers to high-stakes questions

Kaggle forecasting competitions: An overlooked learning opportunity

Hedging and Approximate Truthfulness in Traditional Forecasting Competitions

The M6 forecasting competition: Bridging the gap between forecasting and investment decisions

ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

Forecasting AI Progress: A Research Agenda

The design features of forecasting support systems and their effectiveness

Crowd prediction systems: Markets, polls, and elite forecasters

Four Facets of Forecast Felicity: Calibration, Predictiveness, Randomness and Regret

Predict. Optimize. Revise. On Forecast and Policy Stability in Energy Management Systems

Who will Win the Data Science Competition? Insights from KDD Cup 2019 and Beyond

Fully Automated Correlated Time Series Forecasting in Minutes

Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?

An Integrated Framework for Team Formation and Winner Prediction in the FIRST Robotics Competition: Model, Algorithm, and Analysis

Comparison and Explanation of Forecasting Algorithms for Energy Time Series

Democratic forecast: Small groups predict the future better than individuals and crowds.