Abstract:Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.

Evaluating Real-Time Probabilistic Forecasts With Application to National Basketball Association Outcome Prediction

Real-time forecasting within soccer matches through a Bayesian lens

Calibrated Probabilistic Forecasts for Arbitrary Sequences

Improving Sports Outcome Prediction Process Using Integrating Adaptive Weighted Features and Machine Learning Techniques

Integration of machine learning XGBoost and SHAP models for NBA game outcome prediction and quantitative analysis methodology

Hybrid Basketball Game Outcome Prediction Model by Integrating Data Mining Methods for the National Basketball Association

Nonparametric Stochastic Differential Equations for Ultra-Short-Term Probabilistic Forecasting of Wind Power Generation

Integrating Advanced Statistics to Predict the Outcome of National Basketball Association Games with Machine Learning

Probabilistic recalibration of forecasts

Smart Sports Predictions via Hybrid Simulation: NBA Case Study

Modeling the complexity of basketball games using marked mutually exciting point processes

A compound framework for sports results prediction: A football case study

A Practical Probabilistic Benchmark for AI Weather Models

How often does the best team win? A unified approach to understanding randomness in North American sport

Bayesian estimation of in-game home team win probability for college basketball

Evaluating the effectiveness of machine learning models for performance forecasting in basketball: a comparative study

Modeling and Predicting the Outcomes of NBA Basketball Games

Penalized Regression Models for the NBA

ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

A Continuous-Time Stochastic Process for High-Resolution Network Data in Sports

Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data