Abstract:Severe convective weather events, such as hail, tornadoes, and thunderstorms, often occur quickly yet cause significant damage, costing billions of dollars every year. This highlights the importance of forecasting severe weather threats hours in advance to better prepare meteorologists and residents in at-risk areas. Can modern large foundation models perform such forecasting? Existing weather benchmarks typically focus only on predicting time-series changes in certain weather parameters (e.g., temperature, moisture) with text-only features. In this work, we introduce WeatherQA, the first multimodal dataset designed for machines to reason about complex combinations of weather parameters (a.k.a., ingredients) and predict severe weather in real-world scenarios. The dataset includes over 8,000 (multi-images, text) pairs for diverse severe weather events. Each pair contains rich information crucial for forecasting -- the images describe the ingredients capturing environmental instability, surface observations, and radar reflectivity, and the text contains forecast analyses written by human experts. With WeatherQA, we evaluate state-of-the-art vision language models, including GPT4, Claude3.5, Gemini-1.5, and a fine-tuned Llama3-based VLM, by designing two challenging tasks: (1) multi-choice QA for predicting affected area and (2) classification of the development potential of severe convection. These tasks require deep understanding of domain knowledge (e.g., atmospheric dynamics) and complex reasoning over multimodal data (e.g., interactions between weather parameters). We show a substantial gap between the strongest VLM, GPT4o, and human reasoning. Our comprehensive case study with meteorologists further reveals the weaknesses of the models, suggesting that better training and data integration are necessary to bridge this gap. WeatherQA link: <a class="link-external link-https" href="https://github.com/chengqianma/WeatherQA" rel="external noopener nofollow">this https URL</a>.

A Machine Learning Approach to Improve the Usability of Severe Thunderstorm Wind Reports

Using Machine Learning to Calibrate Storm-Scale Probabilistic Guidance of Severe Weather Hazards in the Warn-on-Forecast System

Storm gust prediction with the integration of machine learning algorithms and WRF model variables for the Northeast United States

Machine Learning Approaches for Wind Speed Forecasting Using Long-Term Monitoring Data: a Comparative Study

Validating Deep-Learning Weather Forecast Models on Recent High-Impact Extreme Events

Machine learning based automated identification of thunderstorms from anemometric records using shapelet transform

Machine Learning Investigation of Downburst Prone Environments in Canada

Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules

Predictive modeling of severe weather impact on individuals and populations using Machine Learning

Storm Detection by Visual Learning Using Satellite Images

Statistical Modelling of 2m Temperature and 10m Wind Speed Forecast Errors

Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán

Evaluating the Utility of Selected Machine Learning Models for Predicting Stormwater Levels in Small Streams

Integrating machine learning for enhanced wildfire severity prediction: A study in the Upper Colorado River basin

Machine learning motivated data imputation of storm data used in coastal hazard assessments

WeatherQA: Can Multimodal Language Models Reason about Severe Weather?

NOAA ProbSevere v2.0—ProbHail, ProbWind, and ProbTor

Robustified extreme learning machine regression with applications in outlier-blended wind-speed forecasting

Lightning-Fast Convective Outlooks: Predicting Severe Convective Environments with Global AI-based Weather Models

A machine-learning approach to thunderstorm forecasting through post-processing of simulation data