Abstract:Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method, available on Github, is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts. Epidemics of infectious disease cause millions of deaths worldwide each year, and reliable epidemic forecasts can allow public health officials to respond to mitigate the effects of epidemics. However, because epidemic forecasting is a difficult task, many epidemic forecasts are not calibrated. Calibration is a desired property of any forecast, and we provide a post-processing method that recalibrates forecasts. We demonstrate the effectiveness of this method in improving accuracy and calibration on a wide variety of influenza forecasters. We also show a quantitative relationship between calibration and a forecaster's expected score. Our recalibration method is a tool that any forecaster can use, regardless of model choice, to improve forecast accuracy and reliability. This work provides a bridge between forecasting theory, which rarely deals with applications in domains that are new or have little data, and some recent applications of epidemic forecasting, where forecast calibration is rarely analyzed systematically.

Probabilistic recalibration of forecasts

Recalibrating probabilistic forecasts of epidemics

Calibrated Probabilistic Forecasts for Arbitrary Sequences

Parameter uncertainty in forecast recalibration

U-Calibration: Forecasting for an Unknown Agent

Assessing the calibration of multivariate probabilistic forecasts

Calibrated Forecasting and Persuasion

Non-probabilistic odds and forecasting with imperfect models

On the generation of probabilistic forecasts from deterministic models

Score-based calibration testing for multivariate forecast distributions

Robust recalibration of aggregate probability forecasts using meta-beliefs

Evaluating Real-Time Probabilistic Forecasts With Application to National Basketball Association Outcome Prediction

Probabilistic forecast reconciliation under the Gaussian framework

CRPS-based online learning for nonlinear probabilistic forecast combination

Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization

Scoring Rules and Calibration for Imprecise Probabilities

Four Facets of Forecast Felicity: Calibration, Predictiveness, Randomness and Regret

Creating Probabilistic Forecasts from Arbitrary Deterministic Forecasts using Conditional Invertible Neural Networks

Parity Calibration

Risk-based Calibration for Probabilistic Classifiers

A User-Focused Approach to Evaluating Probabilistic and Categorical Forecasts