Abstract:Data assimilation (DA) combines partial observations with a dynamical model to improve state estimation. Filter-based DA uses only past and present data and is the prerequisite for real-time forecasts. Smoother-based DA exploits both past and future observations. It aims to fill in missing data, provide more accurate estimations, and develop high-quality datasets. However, the standard smoothing procedure requires using all historical state estimations, which is storage-demanding, especially for high-dimensional systems. This paper develops an adaptive-lag online smoother for a large class of complex dynamical systems with strong nonlinear and non-Gaussian features, which has important applications to many real-world problems. The adaptive lag allows the DA to utilize only observations within a nearby window, significantly reducing computational storage. Online lag adjustment is essential for tackling turbulent systems, where temporal autocorrelation varies significantly over time due to intermittency, extreme events, and nonlinearity. Based on the uncertainty reduction in the estimated state, an information criterion is developed to systematically determine the adaptive lag. Notably, the mathematical structure of these systems facilitates the use of closed analytic formulae to calculate the online smoother and the adaptive lag, avoiding empirical tunings as in ensemble-based DA methods. The adaptive online smoother is applied to studying three important scientific problems. First, it helps detect online causal relationships between state variables. Second, its advantage of computational storage is illustrated via Lagrangian DA, a high-dimensional nonlinear problem. Finally, the adaptive smoother advances online parameter estimation with partial observations, emphasizing the role of the observed extreme events in accelerating convergence.

A Note on the Relation Between Balenzela's Algorithm for Two-Filter Formula for Smoothing and Information Filter

Revisiting the Two-Filter Formula for Smoothing for State-Space Models

A Bayesian Filtering Algorithm for Gaussian Mixture Models

Distributed Robust Bayesian Filtering for State Estimation.

Bellman filtering and smoothing for state-space models

Direct Method for Yau Filtering System with Nonlinear Observations

Lie Algebraic Method in Nonlinear Filtering with State Space Dimension Up to 6

Solution to Brockett's Problem on Finite-Dimensional Estimation Algebras of Maximal Rank in Nonlinear Filtering

Linear Filtering System With Arbitrary Initial Conditions

Smoothed State Estimation via Efficient Solution of Linear Equations

Maximum likelihood smoothing estimation in state-space models: An incomplete-information based approach

Smoothing Dynamic Systems with State-Dependent Covariance Matrices

Auxiliary two-filter particle smoothing for one generalized hidden Markov model

Direct Method Without the Riccati Equation for a Linear Filtering System with Arbitrary Initial Conditions

Outlier-robust Kalman Filtering through Generalised Bayes

Convergence of sequential quasi-Monte Carlo smoothing algorithms

Robust Bayesian Filtering and Smoothing Using Student's t Distribution

An Adaptive Online Smoother with Closed-Form Solutions and Information-Theoretic Lag Selection for Conditional Gaussian Nonlinear Systems

Explicit Construction of Finite Dimensional Nonlinear Filters with State Space Dimension 2

Finite Dimensional Estimation Algebra for Time-Varying Filtering System and Optimal Transport Particle Filter: A Tangent Flow Point of View

Solution of Filtering Problem with Nonlinear Observations