Abstract:Abstract The evaluation of probabilistic forecasts plays a central role both in the interpretation and in the use of forecast systems and their development. Probabilistic scores (scoring rules) provide statistical measures to assess the quality of probabilistic forecasts. Often, many probabilistic forecast systems are available while evaluations of their performance are not standardized, with different scoring rules being used to measure different aspects of forecast performance. Even when the discussion is restricted to strictly proper scoring rules, there remains considerable variability between them; indeed strictly proper scoring rules need not rank competing forecast systems in the same order when none of these systems are perfect. The locality property is explored to further distinguish scoring rules. The nonlocal strictly proper scoring rules considered are shown to have a property that can produce “unfortunate” evaluations, particularly the fact that the continuous rank probability score prefers the outcome close to the median of the forecast distribution regardless of the probability mass assigned to the value at/near the median raises concern to its use. The only local strictly proper scoring rule, the logarithmic score, has direct interpretations in terms of probabilities and bits of information. The nonlocal strictly proper scoring rules, on the other hand, lack meaningful direct interpretation for decision support. The logarithmic score is also shown to be invariant under smooth transformation of the forecast variable, while the nonlocal strictly proper scoring rules considered may, however, change their preferences due to the transformation. It is therefore suggested that the logarithmic score always be included in the evaluation of probabilistic forecasts.

Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval

Eliciting interval beliefs: An experimental study

Valid prediction intervals for regression problems

Enhancing reliability in prediction intervals using point forecasters: Heteroscedastic Quantile Regression and Width-Adaptive Conformal Inference

Chance Constrained Extreme Learning Machine for Nonparametric Prediction Intervals of Wind Power Generation

Tight Prediction Intervals Using Expanded Interval Minimization

An All-Batch Loss for Constructing Prediction Intervals

Cost-Oriented Prediction Intervals: on Bridging the Gap Between Forecasting and Decision

Beyond Strictly Proper Scoring Rules: The Importance of Being Local

Sparse Interval-valued Time Series Modeling with Machine Learning

Log-time Prediction Markets for Interval Securities

Optimal Nonparametric Prediction Intervals of Electricity Load

Scoring Rules and Calibration for Imprecise Probabilities

Robust Elicitable Functionals

Point forecasting and forecast evaluation with generalized Huber loss

Locally tail-scale invariant scoring rules for evaluation of extreme value forecasts

Adaptive, Distribution-Free Prediction Intervals for Deep Networks

Local scale invariance and robustness of proper scoring rules

Optimal intervals for the scaled-uniform model

An Adaptive Bilevel Programming Model for Nonparametric Prediction Intervals of Wind Power Generation

Prediction intervals for economic fixed-event forecasts