Abstract:Summary Operational earthquake forecasting for risk management and communication during seismic sequences depends on our ability to select an optimal forecasting model. To do this, we need to compare the performance of competing models in prospective experiments, and to rank their performance according to the outcome using a fair, reproducible, and reliable method, usually in a low-probability environment. The Collaboratory for the Study of Earthquake Predictability (CSEP) conducts prospective earthquake forecasting experiments around the globe. In this framework, it is crucial that the metrics employed to rank the competing forecasts are ‘proper’, meaning that, on average, they prefer the data generating model. We prove that the Parimutuel Gambling score, proposed, and in some cases applied, as a metric for comparing probabilistic seismicity forecasts, is in general ‘improper’. In the special case where it is proper, we show it can still be used improperly. We demonstrate the conclusions both analytically and graphically providing a set of simulation based techniques that can be used to assess if a score is proper or not. They only require a data generating model and, at least two forecasts to be compared. We compare the Parimutuel Gambling score’s performance with two commonly-used proper scores (the Brier and logarithmic scores) using confidence intervals to account for the uncertainty around the observed score difference. We suggest that using confidence intervals enables a rigorous approach to distinguish between the predictive skills of candidate forecasts, in addition to their rankings. Our analysis shows that the Parimutuel Gambling score is biased, and the direction of the bias depends on the forecasts taking part in the experiment. Our findings suggest the Parimutuel Gambling score should not be used to distinguishing between multiple competing forecasts, and for care to be taken in the case where only two are being compared.

Evaluation of probabilistic forecasts of binary events with the Neighborhood Brier Divergence Skill Score

Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts

Sequential Aggregation of Probabilistic Forecasts -- Applicaton to Wind Speed Ensemble Forecasts

Beyond Strictly Proper Scoring Rules: The Importance of Being Local

A User-Focused Approach to Evaluating Probabilistic and Categorical Forecasts

Evaluating Real-Time Probabilistic Forecasts With Application to National Basketball Association Outcome Prediction

A Practical Probabilistic Benchmark for AI Weather Models

Evaluating Probabilistic Forecasts with scoringRules

Ranking earthquake forecasts using proper scoring rules: Binary events in a low probability environment

Short-term Probabilistic Forecasts for Direct Normal Irradiance

Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions

Using New Neighborhood-Based Intensity-Scale Verification Metrics to Evaluate WRF Precipitation Forecasts at 4 and 12 Km Grid Spacings

Evaluating Density Forecasts Using Weighted Multivariate Scores in a Risk Management Context

Probabilistic quantitative precipitation field forecasting using a two-stage spatial model

Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation

Why do some probabilistic forecasts lack reliability?

Assessing the calibration of multivariate probabilistic forecasts

Probabilistic Neural Networks for Ensemble Postprocessing

Evaluating the Discrimination Ability of Proper Multivariate Scoring Rules

Weighted scoringRules: Emphasising Particular Outcomes when Evaluating Probabilistic Forecasts

Ensemble weather forecast post-processing with a flexible probabilistic neural network approach