Abstract:Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method, available on Github, is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts. Epidemics of infectious disease cause millions of deaths worldwide each year, and reliable epidemic forecasts can allow public health officials to respond to mitigate the effects of epidemics. However, because epidemic forecasting is a difficult task, many epidemic forecasts are not calibrated. Calibration is a desired property of any forecast, and we provide a post-processing method that recalibrates forecasts. We demonstrate the effectiveness of this method in improving accuracy and calibration on a wide variety of influenza forecasters. We also show a quantitative relationship between calibration and a forecaster's expected score. Our recalibration method is a tool that any forecaster can use, regardless of model choice, to improve forecast accuracy and reliability. This work provides a bridge between forecasting theory, which rarely deals with applications in domains that are new or have little data, and some recent applications of epidemic forecasting, where forecast calibration is rarely analyzed systematically.

Robust recalibration of aggregate probability forecasts using meta-beliefs

Probabilistic recalibration of forecasts

Calibrated Probabilistic Forecasts for Arbitrary Sequences

Calibrated Forecasting and Persuasion

The Surprising Benefits of Base Rate Neglect in Robust Aggregation

U-Calibration: Forecasting for an Unknown Agent

Algorithmic Robust Forecast Aggregation

Calibrated Regression Against An Adversary Without Regret

Recalibrating probabilistic forecasts of epidemics

Leveraging Contextual Counterfactuals Toward Belief Calibration

Learning of Optimal Forecast Aggregation in Partial Evidence Environments

Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration

Robust Calibration For Improved Weather Prediction Under Distributional Shift

Online aggregation of probability forecasts with confidence

Cautious Calibration in Binary Classification

Confidence-Aware Multi-Field Model Calibration

Progress in Absorber R&D 2: Windows

Eliminating Public Knowledge Biases in Small Group Predictions

Parameter uncertainty in forecast recalibration

Optimal reconciliation with immutable forecasts

Robust Decision Aggregation with Adversarial Experts