Abstract:SIAM/ASA Journal on Uncertainty Quantification, Volume 9, Issue 2 , Page 681-707, January 2021. Model error covariances play a central role in the performance of data assimilation methods applied to nonlinear state-space models. However, these covariances are largely unknown in most of the applications. A misspecification of the model error covariance has a strong impact on the computation of the posterior probability density function, leading to unreliable estimations and even to a total failure of the assimilation procedure. In this work, we propose the combination of the expectation maximization (EM) algorithm with an efficient particle filter to estimate the model error covariance using a batch of observations. Based on the EM algorithm principles, the proposed method encompasses two stages: the expectation stage, in which a particle filter is used with the present updated value of the model error covariance as given to find the probability density function that maximizes the likelihood, followed by a maximization stage, in which the expectation under the probability density function found in the expectation step is maximized as a function of the elements of the model error covariance. This novel algorithm here presented combines the EM algorithm with a fixed point algorithm and does not require a particle smoother to approximate the posterior densities. We demonstrate that the new method accurately and efficiently solves the linear model problem. Furthermore, for the chaotic nonlinear Lorenz-96 model the method is stable even for observation error covariance 10 times larger than the estimated model error covariance matrix and also is successful in moderately large dimensional situations where the dimension of the estimated matrix is 40 x 40.

On building the state error covariance from a state estimate

Model error covariance estimation in particle and ensemble Kalman filters using an online expectation-maximization algorithm

Observation Error Covariance Specification in Dynamical Systems for Data assimilation using Recurrent Neural Networks

Estimating correlated observation error statistics using an ensemble transform Kalman filter

Model uncertainty estimation using the expectation maximization algorithm and a particle flow filter

A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

Model Error Estimation Using the Expectation Maximization Algorithm and a Particle Flow Filter

Online machine-learning forecast uncertainty estimation for sequential data assimilation

On the Impact of Sampling on Deep Sequential State Estimation

Adaptive Error Covariances Estimation Methods for Ensemble Kalman Filters

Lyapunov-Based Reinforcement Learning State Estimator

Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components

Controlling overestimation of error covariance in ensemble Kalman filters with sparse observations: A variance limiting Kalman filter

Estimating Model Error Covariances with Artificial Neural Networks

Optimality of Variational Data Assimilation and Its Relationship with the Kalman Filter and Smoother

Uniform Error Bounds of the Ensemble Transform Kalman Filter for Chaotic Dynamics with Multiplicative Covariance Inflation

Exact nonlinear state estimation

On‐line machine‐learning forecast uncertainty estimation for sequential data assimilation

Deep learning-based sequential data assimilation for chaotic dynamics identifies local instabilities from single state forecasts

A fast, single-iteration ensemble Kalman smoother for sequential data assimilation