Velma K. Lopez,Estee Y. Cramer,Robert Pagano,John M. Drake,Eamon B. O’Dea,Madeline Adee,Turgay Ayer,Jagpreet Chhatwal,Ozden O. Dalgic,Mary A. Ladd,Benjamin P. Linas,Peter P. Mueller,Jade Xiao,Johannes Bracher,Alvaro J. Castro Rivadeneira,Aaron Gerding,Tilmann Gneiting,Yuxin Huang,Dasuni Jayawardena,Abdul H. Kanji,Khoa Le,Anja Mühlemann,Jarad Niemi,Evan L. Ray,Ariane Stark,Yijin Wang,Nutcha Wattanachit,Martha W. Zorn,Sen Pei,Jeffrey Shaman,Teresa K. Yamana,Samuel R. Tarasewicz,Daniel J. Wilson,Sid Baccam,Heidi Gurung,Steve Stage,Brad Suchoski,Lei Gao,Zhiling Gu,Myungjin Kim,Xinyi Li,Guannan Wang,Lily Wang,Yueying Wang,Shan Yu,Lauren Gardner,Sonia Jindal,Maximilian Marshall,Kristen Nixon,Juan Dent,Alison L. Hill,Joshua Kaminsky,Elizabeth C. Lee,Joseph C. Lemaitre,Justin Lessler,Claire P. Smith,Shaun Truelove,Matt Kinsey,Luke C. Mullany,Kaitlin Rainwater-Lovett,Lauren Shin,Katharine Tallaksen,Shelby Wilson,Dean Karlen,Lauren Castro,Geoffrey Fairchild,Isaac Michaud,Dave Osthus,Jiang Bian,Wei Cao,Zhifeng Gao,Juan Lavista Ferres,Chaozhuo Li,Tie-Yan Liu,Xing Xie,Shun Zhang,Shun Zheng,Matteo Chinazzi,Jessica T. Davis,Kunpeng Mu,Ana Pastore y Piontti,Alessandro Vespignani,Xinyue Xiong,Robert Walraven,Jinghui Chen,Quanquan Gu,Lingxiao Wang,Pan Xu,Weitong Zhang,Difan Zou,Graham Casey Gibson,Daniel Sheldon,Ajitesh Srivastava,Aniruddha Adiga,Benjamin Hurt,Gursharn Kaur,Bryan Lewis,Madhav Marathe,Akhil Sai Peddireddy,Przemyslaw Porebski,Srinivasan Venkatramanan,Lijing Wang,Pragati V. Prasad,Jo W. Walker,Alexander E. Webber,Rachel B. Slayton,Matthew Biggerstaff,Nicholas G. Reich,Michael A. Johansson

Abstract:During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1–4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making. As SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.

Can auxiliary indicators improve COVID-19 forecasting and hotspot prediction?

A Prediction Model Based on Machine Learning for Diagnosing the Early COVID-19 Patients

Assessing the utility of COVID-19 case reports as a leading indicator for hospitalization forecasting in the United States

Empirical Quantitative Analysis of COVID-19 Forecasting Models

A Comprehensive Statistical Analysis of COVID-19 Trends: Global and U.S. Insights through ARIMA, Regression, and Spatial Models

Challenges of COVID-19 Case Forecasting in the US, 2020–2021

Forecasting local surges in COVID-19 hospitalizations through adaptive decision tree classifiers

A Study of Data-driven Methods for Adaptive Forecasting of COVID-19 Cases

Prediction of COVID-19 Data Using Hybrid Modeling Approaches

Enhancing COVID-19 Case Forecasting in the United States: A Comparative Analysis of ARIMA, SARIMA, and RNN Models with Grid Search Optimization

All Models Are Useful: Bayesian Ensembling for Robust High Resolution COVID-19 Forecasting

A Regionally Tailored Epidemiological Forecast and Monitoring Program to Guide a Healthcare System in the COVID-19 Pandemic

COVID-19 forecasts via stock market indicators

Monitoring and Forecasting COVID-19: Statistical Heuristic Regression, Susceptible-Infected-Removed model and, Spatial Stochastics

COVID-19 hospitalizations forecasts using internet search data

A Google–Wikipedia–Twitter Model as a Leading Indicator of the Numbers of Coronavirus Deaths

Joint COVID-19 and influenza-like illness forecasts in the United States using internet search information

Forecasting the spread of COVID-19 based on policy, vaccination, and Omicron data

A Hierarchical Bayesian Model for Stochastic Spatiotemporal SIR Modeling and Prediction of COVID-19 Cases and Hospitalizations

Predictive healthcare modeling for early pandemic assessment leveraging deep auto regressor neural prophet

The disutility of compartmental model forecasts during the COVID-19 pandemic