Abstract:Remote sensing data play a critical role in improving numerical weather prediction (NWP). However, the physical principles of radiation dictate that data voids frequently exist in physical space (e.g., subcloud area for satellite infrared radiance or no-precipitation region for radar reflectivity). Such data gaps impair the accuracy of initial conditions derived from data assimilation (DA), which has a negative impact on NWP. We use the barotropic vorticity equation to demonstrate the potential of deep learning augmented data assimilation (DDA), which involves reconstructing spatially complete pseudo-observation fields from incomplete observations and using them for DA. By training a convolutional autoencoder (CAE) with a long simulation at a coarse "forecast" resolution (T63), we obtained a deep learning approximation of the "reconstruction operator," which maps spatially incomplete observations to a model state with full spatial coverage and resolution. The CAE was applied to an incomplete streamfunction observation (similar to 30% missing) from a high-resolution benchmark simulation and demonstrated satisfactory reconstruction performance, even when only very sparse (1/16 of T63 grid density) observations were used as input. When only spatially incomplete observations are used, the analysis fields obtained from ensemble square root filter (EnSRF) assimilation exhibit significant error. However, in DDA, when EnSRF takes in the combined data from the incomplete observations and CAE reconstruction, analysis error reduces significantly. Such gains are more pronounced with sparse observation and small ensemble size because the DDA analysis is much less sensitive to observation density and ensemble size than the conventional DA analysis, which is based solely on incomplete observations. Significance StatementData assimilation plays a critical role in improving the skills of modern numerical weather prediction by establishing accurate initial conditions. However, unobservable regions are common in observation data, particularly those derived from remote sensing. The nonlinear relationship between data from observable regions and the physical state of unobservable regions may impede DA efficiency. As a result, we propose that deep learning be used to improve data assimilation in such cases by reconstructing a spatially complete first guess of the physical state with deep learning and then applying data assimilation to the reconstructed field. Such deep learning augmentation is found effective in improving the accuracy of data assimilation, especially for sparse observation and small ensemble size.

"Big Data Assimilation" Toward Post-Petascale Severe Weather Prediction: An Overview and Progress.

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5

Data Assimilation in the Geosciences - An overview on methods, issues and perspectives

ADAF: An Artificial Intelligence Data Assimilation Framework for Weather Forecasting

Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK

A Data-Driven Method for Hybrid Data Assimilation with Multilayer Perceptron

Survey of data assimilation methods for convective‐scale numerical weather prediction at operational centres

Exascale Computing and Data Handling: Challenges and Opportunities for Weather and Climate Prediction

Assessing the Feasibility of an NWP Satellite Data Assimilation System Entirely Based on AI Techniques

Data Assimilation in Terrestrial Hydrology Based on Deep Learning Fusing Remote Sensing Big Data:Research Advances and Key Scientific Issues

Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations

Data-Driven Weather Forecasting and Climate Modeling from the Perspective of Development

Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales

A Scalable Real-Time Data Assimilation Framework for Predicting Turbulent Atmosphere Dynamics

A Hybrid Data Assimilation System Based on Machine Learning

Deep Learning Augmented Data Assimilation: Reconstructing Missing Information with Convolutional Autoencoders

Dynamic Meteorology: Data Assimilation Methods

Atmosphere and sea surface temperature coupled data assimilation with enhanced use of SST-sensitive microwave radiances

The Importance of Data Assimilation Components for Initial Conditions and Subsequent Error Growth

Development of the Real‐Time 30‐s‐Update Big Data Assimilation System for Convective Rainfall Prediction with a Phased Array Weather Radar: Description and Preliminary Evaluation

The Theory and Research Progress of Hybrid Data Assimilation