Abstract:Remote sensing data play a critical role in improving numerical weather prediction (NWP). However, the physical principles of radiation dictate that data voids frequently exist in physical space (e.g., subcloud area for satellite infrared radiance or no-precipitation region for radar reflectivity). Such data gaps impair the accuracy of initial conditions derived from data assimilation (DA), which has a negative impact on NWP. We use the barotropic vorticity equation to demonstrate the potential of deep learning augmented data assimilation (DDA), which involves reconstructing spatially complete pseudo-observation fields from incomplete observations and using them for DA. By training a convolutional autoencoder (CAE) with a long simulation at a coarse "forecast" resolution (T63), we obtained a deep learning approximation of the "reconstruction operator," which maps spatially incomplete observations to a model state with full spatial coverage and resolution. The CAE was applied to an incomplete streamfunction observation (similar to 30% missing) from a high-resolution benchmark simulation and demonstrated satisfactory reconstruction performance, even when only very sparse (1/16 of T63 grid density) observations were used as input. When only spatially incomplete observations are used, the analysis fields obtained from ensemble square root filter (EnSRF) assimilation exhibit significant error. However, in DDA, when EnSRF takes in the combined data from the incomplete observations and CAE reconstruction, analysis error reduces significantly. Such gains are more pronounced with sparse observation and small ensemble size because the DDA analysis is much less sensitive to observation density and ensemble size than the conventional DA analysis, which is based solely on incomplete observations. Significance StatementData assimilation plays a critical role in improving the skills of modern numerical weather prediction by establishing accurate initial conditions. However, unobservable regions are common in observation data, particularly those derived from remote sensing. The nonlinear relationship between data from observable regions and the physical state of unobservable regions may impede DA efficiency. As a result, we propose that deep learning be used to improve data assimilation in such cases by reconstructing a spatially complete first guess of the physical state with deep learning and then applying data assimilation to the reconstructed field. Such deep learning augmentation is found effective in improving the accuracy of data assimilation, especially for sparse observation and small ensemble size.

Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5

Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales

Advancing Data-driven Weather Forecasting: Time-Sliding Data Augmentation of ERA5

Global atmospheric data assimilation with multi-modal masked autoencoders

A Hybrid Data‐Driven and Data Assimilation Method for Spatiotemporal Forecasting: PM2.5 Forecasting in China

A Machine Learning Augmented Data Assimilation Method for High‐Resolution Observations

ADAF: An Artificial Intelligence Data Assimilation Framework for Weather Forecasting

Data Assimilation with Machine Learning Surrogate Models: A Case Study with FourCastNet

Survey of data assimilation methods for convective‐scale numerical weather prediction at operational centres

Data driven weather forecasts trained and initialised directly from observations

Data-Driven Weather Forecasting and Climate Modeling from the Perspective of Development

DiffDA: a Diffusion Model for Weather-scale Data Assimilation

Exploring the Use of Machine Learning Weather Models in Data Assimilation

Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations

Deep Learning Augmented Data Assimilation: Reconstructing Missing Information with Convolutional Autoencoders

FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation

Exploring the integration of a global AI model with traditional data assimilation in weather forecasting

A Benchmark for AI-based Weather Data Assimilation

Oceanic Data Assimilation Study in Northern Chile: Use of a 3DVAR Method

Assimilating Retrieved Water Vapor and Radar Data From NCAR S-PolKa: Performance and Validation Using Real Cases