Abstract:Background: In the past, two studies found ensembles of human judgement forecasts of COVID-19 to show predictive performance comparable to ensembles of computational models, at least when predicting case incidences. We present a follow-up to a study conducted in Germany and Poland and investigate a novel joint approach to combine human judgement and epidemiological modelling. Methods: From May 24th to August 16th 2021, we elicited weekly one to four week ahead forecasts of cases and deaths from COVID-19 in the UK from a crowd of human forecasters. A median ensemble of all forecasts was submitted to the European Forecast Hub. Participants could use two distinct interfaces: in one, forecasters submitted a predictive distribution directly, in the other forecasters instead submitted a forecast of the effective reproduction number R t. This was then used to forecast cases and deaths using simulation methods from the EpiNow2 R package. Forecasts were scored using the weighted interval score on the original forecasts, as well as after applying the natural logarithm to both forecasts and observations. Results: The ensemble of human forecasters overall performed comparably to the official European Forecast Hub ensemble on both cases and deaths, although results were sensitive to changes in details of the evaluation. R t forecasts performed comparably to direct forecasts on cases, but worse on deaths. Self-identified "experts" tended to be better calibrated than "non-experts" for cases, but not for deaths. Conclusions: Human judgement forecasts and computational models can produce forecasts of similar quality for infectious disease such as COVID-19. The results of forecast evaluations can change depending on what metrics are chosen and judgement on what does or doesn't constitute a "good" forecast is dependent on the forecast consumer. Combinations of human and computational forecasts hold potential but present real-world challenges that need to be solved.

Use of heuristics: Insights from forecasting research

Judgment under Uncertainty: Heuristics and Biases

Heuristic assumptions

Asking Better Questions -- The Art and Science of Forecasting: A mechanism for truer answers to high-stakes questions

Light-touch forecasting: A novel method to combine human judgment with statistical algorithms

Affective forecasting as an adaptive learning process.

How heuristics in judgement influence the securities investment decision process

Judgmental selection of forecasting models

Heuristic decision-making across adulthood.

The Impact of Heuristics Bias on Investment Decision: An Empirical Study

Integrating human judgement into quantitative forecasting methods: A review

Heuristics in sport: A scoping review

Forecasting in social settings: The state of the art

How to evaluate the rationality of heuristics?

Visualizing Uncertainty in Time Series Forecasts: The Impact of Uncertainty Visualization on Users' Confidence, Algorithmic Advice Utilization, and Forecasting Performance

Forecast and hedging of risks in decision-making by an investigator and a judge

ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

The limits of forecasting methods in anticipating rare events

A decision-congruent heuristic gives superior metacognitive sensitivity under realistic variance assumptions.

Human judgement forecasting of COVID-19 in the UK