Abstract:We discuss an approach to probabilistic forecasting based on two chained machine-learning steps: a dimensional reduction step that learns a reduction map of predictor information to a low-dimensional space in a manner designed to preserve information about forecast quantities; and a density estimation step that uses the probabilistic machine learning technique of normalizing flows to compute the joint probability density of reduced predictors and forecast quantities. This joint density is then renormalized to produce the conditional forecast distribution. In this method, probabilistic calibration testing plays the role of a regularization procedure, preventing overfitting in the second step, while effective dimensional reduction from the first step is the source of forecast sharpness. We verify the method using a 22-year 1-hour cadence time series of Weather Research and Forecasting (WRF) simulation data of surface wind on a grid.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper aims to address the issue of probabilistic forecasting in weather prediction. Specifically, the authors propose a deep learning-based approach to achieve probabilistic forecasting through two consecutive machine learning steps: 1. **Dimensionality Reduction Step**: First, a dimensionality reduction mapping is learned to map high-dimensional predictive information to a low-dimensional space while retaining as much information about the forecast quantity as possible. 2. **Density Estimation Step**: Then, using the probabilistic machine learning technique of Normalizing Flows, the joint probability density of the reduced-dimensional predictive information and the forecast quantity is calculated. By re-normalizing this joint density, the conditional forecast distribution is obtained. ### Main Challenges 1. **High-Dimensional Data Processing**: Weather forecast data typically has high dimensionality, making direct probabilistic forecasting computationally challenging. 2. **Information Retention**: During the dimensionality reduction process, it is necessary to retain as much information related to the forecast quantity as possible. 3. **Model Calibration**: The forecast model needs to be calibrated to ensure that its probability distribution accurately reflects the uncertainty in the actual data. ### Method Overview 1. **Dimensionality Reduction Step**: - A neural network is used to learn a dimensionality reduction mapping \( T \), which maps high-dimensional predictive information \( X \) to a low-dimensional representation \( T(X) \). - The optimal dimensionality reduction mapping \( T \) is selected by maximizing the mutual information \( I(Y; T(X)) \). 2. **Density Estimation Step**: - The joint distribution \( P(Y, T(X)) \) is modeled using the Normalizing Flows technique. - By re-normalizing the joint distribution, the conditional forecast distribution \( P(Y | T(X)) \) is obtained. ### Validation Method The authors validated the proposed method using a 22-year, hourly time series dataset of surface wind speed grid data simulated by the Weather Research and Forecasting (WRF) model. ### Conclusion The method, through effective dimensionality reduction and non-parametric density estimation, can improve forecast accuracy while maintaining model calibration. This provides a new effective approach for probabilistic forecasting in weather prediction.

A Deep Learning Approach to Probabilistic Forecasting of Weather

A Novel Sequence-to-Sequence-Based Deep Learning Model for Multistep Load Forecasting.

Leveraging deterministic weather forecasts for in-situ probabilistic temperature predictions via deep learning

Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows

Ensemble Deep Learning-Based Non-Crossing Quantile Regression for Nonparametric Probabilistic Forecasting of Wind Power Generation

A Deep Learning-Based Weather Forecast System for Data Volume and Recency Analysis

Nonparametric Conditional Density Estimation In A Deep Learning Framework For Short-Term Forecasting

A framework for probabilistic weather forecast post-processing across models and lead times using machine learning

Quantifying the Uncertainty of Precipitation Forecasting Using Probabilistic Deep Learning

Improving seasonal forecast using probabilistic deep learning

Data driven weather forecasts trained and initialised directly from observations

Deep Learning for Post-Processing Ensemble Weather Forecasts

Probabilistic multi-step ahead streamflow forecast based on deep learning

Deep Learning of a 200-member Ensemble with a Limited Historical Training to Improve the Prediction of Extreme Precipitation Events

Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales

Deep probabilistic time series forecasting using augmented recurrent input for dynamic systems

Deep Learning-Based Weather Prediction: A Survey

Weather Prediction with Diffusion Guided by Realistic Forecast Processes