Abstract:This paper examines learning the optimal filtering policy, known as the Kalman gain, for a linear system with unknown noise covariance matrices using noisy output data. The learning problem is formulated as a stochastic policy optimization problem, aiming to minimize the output prediction error. This formulation provides a direct bridge between data-driven optimal control and, its dual, optimal filtering. Our contributions are twofold. Firstly, we conduct a thorough convergence analysis of the stochastic gradient descent algorithm, adopted for the filtering problem, accounting for biased gradients and stability constraints. Secondly, we carefully leverage a combination of tools from linear system theory and high-dimensional statistics to derive bias-variance error bounds that scale logarithmically with problem dimension, and, in contrast to subspace methods, the length of output trajectories only affects the bias term.

What problem does this paper attempt to address?

The problem this paper attempts to address is how to learn the optimal filtering strategy (i.e., Kalman gain) using noisy output data in linear systems when the noise covariance matrix is unknown. Specifically, the paper formulates this learning problem as a stochastic policy optimization problem with the objective of minimizing the output prediction error. Through this approach, the paper establishes a direct bridge from data-driven optimal control to optimal filtering. ### Main Contributions 1. **Convergence Analysis**: The paper provides a detailed convergence analysis of the stochastic gradient descent algorithm used for the filtering problem, considering biased gradients and stability constraints. 2. **Error Bound**: Using linear system theory and high-dimensional statistical tools, the paper derives bias-variance error bounds that are logarithmic in the problem dimension. Unlike subspace methods, the length of the output trajectory only affects the bias term. ### Problem Background In system theory, the duality between control and estimation plays a crucial role, connecting two different synthesis problems. This duality provides an effective bridge for the development of theoretical and computational techniques. For example, the stability proof of the Kalman filter relies on its dual problem—the stability properties of the optimal feedback gain in the Linear Quadratic Regulator (LQR) optimal control problem. ### Learning Setup The paper considers a system with known linear dynamics and observation models but unknown process and measurement noise covariances. The goal is to learn the optimal steady-state Kalman gain from a training dataset containing independent observation signal realizations. This problem has a rich history in the field of adaptive Kalman filtering, with common solutions including Bayesian inference, maximum likelihood, covariance matching, and innovation correlation methods. ### Methodology 1. **Stochastic Optimization Problem**: An alternative objective function for the estimation error is defined, which minimizes the prediction error of the observation \( y(T) \). 2. **Stochastic Gradient Descent Algorithm**: A stochastic gradient descent (SGD) algorithm is proposed to learn the optimal Kalman gain, providing new non-asymptotic error guarantees considering biased gradients and stability constraints. ### Theoretical Results - **Linear Convergence**: Assuming the system is observable and the dynamics and measurement noise are bounded, the stochastic gradient descent algorithm with small step sizes for direct policy updates globally converges linearly with high probability to the optimal steady-state Kalman gain. - **Error Bound**: By analyzing the bias-variance error of the gradient estimation, sufficient batch sizes and trajectory lengths are provided to ensure the required gradient estimation error bounds. ### Experimental Validation The paper provides numerical results in the appendix, validating the effectiveness of the proposed algorithm. ### Conclusion By leveraging the duality between control and estimation and the latest advances in data-driven optimal control, this paper offers new insights and methods for solving the optimal filtering problem under unknown noise covariance. These methods not only have rigorous non-asymptotic error guarantees in theory but also demonstrate good performance in practical applications.

Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances

Data-Based Filters for Non-Gaussian Dynamic Systems with Unknown Output Noise Covariance

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Data-Driven Kalman Filter using Maximum Likelihood Optimization

An Extended Kalman Filter for Data-enabled Predictive Control

Approximate Optimal Filter for Linear Gaussian Time-invariant Systems

Optimal Linear Filtering for Discrete-Time Systems with Infinite-Dimensional Measurements

Variational Nonlinear Kalman Filtering with Unknown Process Noise Covariance

Data-Driven Deep Learning Based Feedback Linearization of Systems with Unknown Dynamics

Robust Kalman filters with unknown covariance of multiplicative noise

An Adaptive State Filtering Algorithm for Systems With Partially Known Dynamics

Mean-square error constrained approach to robust stochastic iterative learning control

Joint State Estimation and Noise Identification Based on Variational Optimization

Kalman Filtering Through the Feedback Adaption of Prior Error Covariance.

Learning Optimal Control Policy for Unknown Discrete-Time Systems

Smoothing Dynamic Systems with State-Dependent Covariance Matrices

Variational Bayesian Adaptation of Noise Covariances in Non-Linear Kalman Filtering

Quadratic covariance‐constrained filtering for linear and non‐linear systems with non‐Gaussian noises

An Adaptive Iterated Kalman Filter

Data-driven Kalman Filter with Kernel-based Koopman Operators for Nonlinear Robot Systems.

Robust Optimal Filtering for Linear Time-Varying Systems with Stochastic Uncertainties