Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows

Zhecheng Liu,Diederik Beckers,Jeff D. Eldredge

2024-08-27

Abstract:The intrinsic high dimension of fluid dynamics is an inherent challenge to control of aerodynamic flows, and this is further complicated by a flow's nonlinear response to strong disturbances. Deep reinforcement learning, which takes advantage of the exploratory aspects of reinforcement learning (RL) and the rich nonlinearity of a deep neural network, provides a promising approach to discover feasible control strategies. However, the typical model-free approach to reinforcement learning requires a significant amount of interaction between the flow environment and the RL agent during training, and this high training cost impedes its development and application. In this work, we propose a model-based reinforcement learning (MBRL) approach by incorporating a novel reduced-order model as a surrogate for the full environment. The model consists of a physics-augmented autoencoder, which compresses high-dimensional CFD flow field snaphsots into a three-dimensional latent space, and a latent dynamics model that is trained to accurately predict the long-time dynamics of trajectories in the latent space in response to action sequences. The robustness and generalizability of the model is demonstrated in two distinct flow environments, a pitching airfoil in a highly disturbed environment and a vertical-axis wind turbine in a disturbance-free environment. Based on the trained model in the first problem, we realize an MBRL strategy to mitigate lift variation during gust-airfoil encounters. We demonstrate that the policy learned in the reduced-order environment translates to an effective control strategy in the full CFD environment.

Fluid Dynamics,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the control problem in strongly perturbed and unstable airflows. Specifically, the paper focuses on how to use the Model - Based Reinforcement Learning (MBRL) method to effectively control unstable aerodynamic flows that exhibit highly nonlinear responses under strong external disturbances. Such control is crucial for improving the performance, efficiency, and safety of various systems, such as those in the energy, transportation, and biomedical science fields.

### Main Challenges
1. **High - Dimensional Problem**: The inherently high - dimensional nature of fluid dynamics makes airflow control challenging in itself.
2. **Nonlinear Response**: The nonlinear response of the airflow to strong perturbations further complicates the control problem.
3. **High Training Cost**: Traditional model - free reinforcement learning methods require a large number of interactions between the environment and the agent for training, which is very expensive in the fluid dynamics environment because each interaction may require running a Computational Fluid Dynamics (CFD) simulation.

### Solutions
To address these challenges, the authors propose a Model - Based Reinforcement Learning method (MBRL) by introducing a novel reduced - order model as a substitute for the complete environment. This model consists of two main parts:
- **Physics - Augmented Autoencoder (PA - AE)**: Compresses high - dimensional CFD flow field snapshots into a three - dimensional latent space and is able to reconstruct the flow field state.
- **Latent Dynamics Model (LDM)**: Trained to accurately predict the long - time dynamic response of trajectories in the latent space in response to action sequences.

### Application Scenarios
The authors verified the robustness and generalization ability of the model in two different fluid environments:
1. **Pitching Airfoil in a Strongly Perturbed Environment**: In this environment, the model can effectively mitigate lift changes caused by the encounter between gusts and the airfoil.
2. **Vertical - Axis Wind Turbine in an Unperturbed Environment**: In this environment, the model also shows good control effects.

### Experimental Results
Based on the trained model, the authors implemented the MBRL policy and successfully transformed the policy learned in the reduced - order environment into an effective control policy in the full CFD environment, thus demonstrating the effectiveness and practicality of this method.

### Summary
This paper proposes an efficient and economical Model - Based Reinforcement Learning method for controlling unstable airflows under strong perturbations by combining the Physics - Augmented Autoencoder and the Latent Dynamics Model. This method not only reduces the training cost but also improves the robustness and generalization ability of the control strategy.