Abstract:Statistical sampling, as we know it, works for single variables of the atmosphere. In one dimension (left panel), the distribution of reliable samples (blue curve) overlaps with the distribution of best guesses (black curve), and importantly, contains reality (red vertical line). In multidimensional spaces like resolved‐scale atmospheric dynamics (150 dimensions, right panel), sample points uniformly degrade the best estimate, so they do not contain reality. The attempt at capturing the evolution of the atmosphere with dynamically generated ensembles is futile. Uncertain quantities are often described through statistical samples. Can samples for numerical weather forecasts be generated dynamically? At a great expense, they can. With statistically constrained perturbations, a cloud of initial states is created and then integrated forward in time. By now, this technique has become ubiquitous in weather and climate research and operations. Ensembles are widely used, with demonstrated value. The atmosphere evolves in a multidimensional phase space. Does a cloud of ensemble solutions encompass the evolution of the real atmosphere? Theoretically, random perturbations in high‐dimensional spaces have negligible projection in any direction, including the error in the best estimate, therefore consistently degrading that. As the bulk of the perturbation variance lies in the null space of error, samples in multidimensional space do not contain reality. An evaluation suggests that initial and short‐range forecast error and ensemble perturbations are random draws from a high‐dimensional domain we call the subspace of possible error. Error in any initial condition is partly a result of stochastic observational and assimilation noise, while perturbations explore other, mostly independent directions from the subspace of possible error that may have resulted from other configurations of stochastic noise. What benefits may arise from the deterministic projection of such noise? Consistent with theoretical expectations, ensemble members consistently degrade the skill of the unperturbed forecast until medium range. The mean and all other products derived from ensembles suffer an 18‐hour loss in forecast Information. Since Information is a sufficient statistic, any rational user can benefit more from the unperturbed, than from an ensemble of weather forecasts. Furthermore, case‐dependent variations in the distribution or spread of ensembles have no impact on commonly used metrics. Can alternative, statistical applications provide comparable, or even higher‐quality probabilistic and other products, at the fraction of the cost of running an ensemble?

Generative emulation of weather forecast ensembles with diffusion models

SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models

Latent Diffusion Model for Generating Ensembles of Climate Simulations

Probabilistic Emulation of a Global Climate Model with Spherical DYffusion

Precipitation nowcasting with generative diffusion models

Continuous Ensemble Weather Forecasting with Diffusion models

Deep Learning for Post-Processing Ensemble Weather Forecasts

Diffusion-Based Joint Temperature and Precipitation Emulation of Earth System Models

Computing the ensemble spread from deterministic weather predictions using conditional generative adversarial networks

CoDiCast: Conditional Diffusion Model for Weather Prediction with Uncertainty Quantification

Weather Prediction with Diffusion Guided by Realistic Forecast Processes

On the generation of probabilistic forecasts from deterministic models

Uncertainty quantification for data-driven weather models

Towards replacing precipitation ensemble predictions systems using machine learning

Deterministic Guidance Diffusion Model for Probabilistic Weather Forecasting

Generative Diffusion for Regional Surrogate Models From Sea‐Ice Simulations

Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification

Denoising diffusion probabilistic models for probabilistic energy forecasting

A Generative Diffusion Model for Probabilistic Ensembles of Precipitation Maps Conditioned on Multisensor Satellite Observations

Ensemble forecasting: A foray of dynamics into the realm of statistics

Machine learning emulation of precipitation from km-scale regional climate simulations using a diffusion model