Reinforcement learning-based estimation for partial differential equations

Saviz Mowlavi,Mouhacine Benosman

2024-04-04

Abstract:In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original high-dimensional dynamics onto a computationally tractable low-dimensional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlinearity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier-Stokes equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman filter designed using the same ROM. Moreover, it yields accurate high-dimensional state estimates for trajectories corresponding to various physical parameter values, without direct knowledge of the latter.

Machine Learning,Systems and Control

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to design a state estimator in a system controlled by nonlinear partial differential equations (PDEs) to infer the high - dimensional state of the system from sparse measurement data. Specifically, the paper focuses on continuous systems such as fluid flow. Since these systems are usually described by high - dimensional and nonlinear dynamic models, directly applying traditional state - estimation techniques (such as the Kalman filter) becomes computationally infeasible. Therefore, researchers usually design state estimators based on reduced - order models (ROMs), that is, project the original high - dimensional dynamics onto a low - dimensional space for computational processing. However, a major challenge of this method is that ROMs often provide a simplified and imperfect dynamic description, which will affect the performance of the state estimator.

To solve this problem, the paper proposes a new state - estimator design method - Reinforcement Learning Reduced - Order Estimator (RL - ROE). The core idea of RL - ROE is to use the nonlinear policy obtained through reinforcement learning training to correct the errors brought by measurement data in the state - estimation framework based on ROMs. This method can not only utilize the incomplete dynamic knowledge provided by ROMs, but also effectively compensate for the errors in ROMs, thereby improving the accuracy of state estimation when the number of sensors is very limited. The paper shows the advantages of RL - ROE over the traditional Kalman filter (KF - ROE) through examples of the Burgers equation and the Navier - Stokes equation, especially under sparse measurement conditions, RL - ROE performs better.