U-Net Kalman Filter (UNetKF): An Example of Machine Learning-assisted Ensemble Data Assimilation

Feiyu Lu

2024-03-19

Abstract:Machine learning techniques have seen a tremendous rise in popularity in weather and climate sciences. Data assimilation (DA), which combines observations and numerical models, has great potential to incorporate machine learning and artificial intelligence (ML/AI) techniques. In this paper, we use U-Net, a type of convolutional neutral network (CNN), to predict the localized ensemble covariances for the Ensemble Kalman Filter (EnKF) algorithm. Using a 2-layer quasi-geostrophic model, U-Nets are trained using data from EnKF DA experiments. The trained U-Nets are then used to predict the flow-dependent localized error covariance matrices in U-Net Kalman Filter (UNetKF) experiments, which are compared to traditional 3-dimensional variational (3DVar), ensemble 3DVar (En3DVar) and EnKF methods. The performance of UNetKF can match or exceed that of 3DVar, En3DVar or EnKF. We also demonstrate that trained U-Nets can be transferred to a higher-resolution model for UNetKF implementation, which again performs competitively to 3DVar and EnKF, particularly for small ensemble sizes.

Machine Learning,Atmospheric and Oceanic Physics

What problem does this paper attempt to address?

This paper is primarily dedicated to addressing the problem of Data Assimilation (DA) in Earth system science, particularly focusing on the limitations of the Ensemble Kalman Filter (EnKF). Specifically, the paper attempts to improve traditional ensemble-based data assimilation methods through machine learning techniques. ### Main Issues - **Ensemble Size Limitation**: Due to computational resource constraints, EnKF typically can only use a limited number of ensemble members, which leads to sampling errors, especially in complex high-resolution models. - **Computational Efficiency**: Traditional EnKF methods require substantial computational resources to estimate model uncertainties, particularly when dealing with large-scale Earth system models. ### Solution - **Proposed Method**: The paper introduces a method called the U-Net Kalman Filter (UNetKF), which utilizes U-Net (a type of convolutional neural network) to predict the local ensemble covariance matrices in the EnKF algorithm. - **Objective**: By using U-Net to predict flow-dependent local ensemble error covariance matrices, the aim is to reduce sampling errors in EnKF and improve computational efficiency. ### Experimental Validation - **Model Setup**: The study uses a 2-layer quasi-geostrophic model as a test case. - **Comparison Methods**: UNetKF is compared with traditional 3D Variational (3DVar), Ensemble 3D Variational (En3DVar), and EnKF methods. - **Results**: The experiments show that for small to medium ensemble sizes, the performance of UNetKF can match or even exceed that of 3DVar, En3DVar, or EnKF. Additionally, the trained U-Net can be transferred to higher-resolution models, and its performance remains competitive. In summary, this paper aims to address the challenges faced by existing data assimilation techniques in handling complex Earth system models by integrating machine learning techniques with traditional data assimilation methods.

U-Net Kalman Filter (UNetKF): An Example of Machine Learning-assisted Ensemble Data Assimilation

Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems

A Deep Neural Network-Ensemble Adjustment Kalman Filter and Its Application on Strongly Coupled Data Assimilation

Adaptive Multi-Fidelity Probabilistic Collocation-Based Kalman Filter for Subsurface Flow Data Assimilation: Numerical Modeling and Real-World Experiment

Data Assimilation for Unsaturated Flow Models with Restart Adaptive Probabilistic Collocation Based Kalman Filter

Tests of an Ensemble Kalman Filter for Mesoscale and Regional-Scale Data Assimilation. Part Iii: Comparison with 3dvar in A Real-Data Case Study

Deterministic ensemble Kalman filter based on two localization techniques for mitigating sampling errors with a quasi-geostrophic model

An Iterative Ensemble Kalman Filter for Data Assimilation

Limited-Area Ensemble-Based Data Assimilation

A Hybrid Nudging-Ensemble Kalman Filter Approach to Data Assimilation. Part I: Application in the Lorenz System

Advancing neural network-based data assimilation for large-scale spatiotemporal systems with sparse observations

Particle network EnKF for large-scale data assimilation

Tests of an Ensemble Kalman Filter for Mesoscale and Regional-Scale Data Assimilation. Part Iv: Comparison with 3dvar in A Month-Long Experiment

Particle Network Ensemble Kalman Filter

A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

A Hybrid Nudging-Ensemble Kalman Filter Approach to Data Assimilation. Part Ii: Application in A Shallow-Water Model

Data Assimilation by Artificial Neural Networks for an Atmospheric General Circulation Model: Conventional Observation

Ensemble Kalman method for learning turbulence models from indirect observation data

A Local Ensemble Kalman Filter for Atmospheric Data Assimilation

Water Level Simulation in River Network by Data Assimilation Using Ensemble Kalman Filter

Towards physics-inspired data-driven weather forecasting: integrating data assimilation with a deep spatial-transformer-based U-NET in a case study with ERA5