Abstract:Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning. Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised. Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (pboth <1 × 10-9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10-14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10-7. Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks. Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.

Pachinko Prediction: A Bayesian method for event prediction from social media data

Real-time Detection of Content Polluters in Partially Observable Twitter Networks

On predictability of rare events leveraging social media: a machine learning perspective

Embedding and Predicting the Event at Early Stage

Investigating the Potential of Aggregated Tweets as Surrogate Data for Forecasting Civil Protests

Predicting the Future With Social Media

A Real-Time Method to Predict Social Media Popularity

Social Media Event Prediction using DNN with Feedback Mechanism

Box office prediction based on microblog

A Survey on Societal Event Forecasting with Deep Learning

Early prediction of the duration of protests using probabilistic Latent Dirichlet Allocation and Decision Trees

A Survey of Prediction Using Social Media

Predicting the Popularity of Reddit Posts with AI

Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID events

Automatic crime prediction using events extracted from twitter posts

Predicting Emerging Trends on Social Media by Modeling it as Temporal Bipartite Networks

Forecasting the subway passenger flow under event occurrences with multivariate disturbances

Bayesian Heatmaps: Probabilistic Classification with Multiple Unreliable Information Sources

Real-time forecasting within soccer matches through a Bayesian lens

Real-time Spatio-temporal Event Detection on Geotagged Social Media

A General Method for Event Detection on Social Media