Abstract:Abstract. There is growing interest in data-driven weather prediction (DDWP), e.g., using convolutional neural networks such as U-NET that are trained on data from models or reanalysis. Here, we propose three components, inspired by physics, to integrate with commonly used DDWP models in order to improve their forecast accuracy. These components are (1) a deep spatial transformer added to the latent space of U-NET to capture rotation and scaling transformation in the latent space for spatiotemporal data, (2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and (3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit and feasibility of each component, we use geopotential height at 500 hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the spatial-transformer-based U-NET (U-STN) clearly outperforms the U-NET, e.g., improving the forecast skill by 45 %. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. This DDWP+DA framework substantially benefits from large (O(1000)) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promise; for example, it reduces the average error by factors of 2–3. These results show the benefits and feasibility of these three components, which are flexible and can be used in a variety of DDWP setups. Furthermore, while here we focus on weather forecasting, the three components can be readily adopted for other parts of the Earth system, such as ocean and land, for which there is a rapid growth of data and need for forecast and assimilation.

Deep learning and a changing economy in weather and climate prediction

Can deep learning beat numerical weather prediction?

Advances and prospects of deep learning for medium-range extreme weather forecasting

A Novel Framework for Spatio-Temporal Prediction of Environmental Data Using Deep Learning

Deep learning and process understanding for data-driven Earth system science

Deep learning for spatiotemporal forecasting in Earth system science: a review

DeepPhysiNet: Bridging Deep Learning and Atmospheric Physics for Accurate and Continuous Weather Modeling

A Deep Learning Earth System Model for Stable and Efficient Simulation of the Current Climate

Data driven weather forecasts trained and initialised directly from observations

What if? Numerical weather prediction at the crossroads

Deep learning, deep change? Mapping the evolution and geography of a general purpose technology

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5

Deep Learning-based Extreme Heatwave Forecast

Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes and Atmospheric Dynamics

Machine Learning in Weather Prediction and Climate Analyses—Applications and Perspectives

Short-term precipitation prediction using deep learning

Data-Driven Weather Forecasting and Climate Modeling from the Perspective of Development

Deep Learning for Day Forecasts from Sparse Observations

Robustness of AI-based weather forecasts in a changing climate