Abstract:Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method, available on Github, is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts. Epidemics of infectious disease cause millions of deaths worldwide each year, and reliable epidemic forecasts can allow public health officials to respond to mitigate the effects of epidemics. However, because epidemic forecasting is a difficult task, many epidemic forecasts are not calibrated. Calibration is a desired property of any forecast, and we provide a post-processing method that recalibrates forecasts. We demonstrate the effectiveness of this method in improving accuracy and calibration on a wide variety of influenza forecasters. We also show a quantitative relationship between calibration and a forecaster's expected score. Our recalibration method is a tool that any forecaster can use, regardless of model choice, to improve forecast accuracy and reliability. This work provides a bridge between forecasting theory, which rarely deals with applications in domains that are new or have little data, and some recent applications of epidemic forecasting, where forecast calibration is rarely analyzed systematically.

Calibrated Probabilistic Forecasts for Arbitrary Sequences

Calibrated Regression Against An Adversary Without Regret

Probabilistic recalibration of forecasts

Evaluating Real-Time Probabilistic Forecasts With Application to National Basketball Association Outcome Prediction

On the generation of probabilistic forecasts from deterministic models

A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction

Recalibrating probabilistic forecasts of epidemics

Breaking the $T^{2/3}$ Barrier for Sequential Calibration

Calibrated Forecasting and Persuasion

U-Calibration: Forecasting for an Unknown Agent

APForecast: An Adaptive Forecasting Method for Data Streams

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Online Calibrated and Conformal Prediction Improves Bayesian Optimization

Decision making with dynamic probabilistic forecasts

Non-probabilistic odds and forecasting with imperfect models

Adaptive Ensemble Learning of Spatiotemporal Processes with Calibrated Predictive Uncertainty: A Bayesian Nonparametric Approach

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting

Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamics

Streaming Motion Forecasting for Autonomous Driving

Robust recalibration of aggregate probability forecasts using meta-beliefs