Model-based deep reinforcement learning for accelerated learning from flow simulations

Andre Weiner,Janis Geise

2024-04-10

Abstract:In recent years, deep reinforcement learning has emerged as a technique to solve closed-loop flow control problems. Employing simulation-based environments in reinforcement learning enables a priori end-to-end optimization of the control system, provides a virtual testbed for safety-critical control applications, and allows to gain a deep understanding of the control mechanisms. While reinforcement learning has been applied successfully in a number of rather simple flow control benchmarks, a major bottleneck toward real-world applications is the high computational cost and turnaround time of flow simulations. In this contribution, we demonstrate the benefits of model-based reinforcement learning for flow control applications. Specifically, we optimize the policy by alternating between trajectories sampled from flow simulations and trajectories sampled from an ensemble of environment models. The model-based learning reduces the overall training time by up to $85\%$ for the fluidic pinball test case. Even larger savings are expected for more demanding flow simulations.

Fluid Dynamics,Computational Engineering, Finance, and Science,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address the high computational cost and long time consumption issues in the closed-loop flow control (AFC) problem in fluid mechanics. Specifically, the authors utilize a model-based deep reinforcement learning approach to accelerate the learning process from flow simulations.

#### Main Objectives
1. **Reduce Training Time**: By combining trajectories generated from real flow simulations and environment models, reduce the overall training time.
2. **Improve Data Efficiency**: Use model predictions to generate more training data, thereby improving data usage efficiency.
3. **Validate Method Effectiveness**: Validate the effectiveness of the model in two classic flow control benchmark cases, including flow around a cylinder and fluidic pinball configuration.

#### Background and Challenges
- **High Computational Cost**: Traditional reinforcement learning methods require a large amount of simulation data, which is very time-consuming and expensive in practical applications.
- **Sample Efficiency Issue**: To achieve efficient closed-loop flow control, a method that can quickly learn with limited data is needed.
- **Model Error Handling**: Effectively creating accurate surrogate models and handling errors in these models is one of the main challenges.

Through the above research, the authors hope to demonstrate the potential advantages of model-based reinforcement learning in fluid mechanics and prove that this approach can significantly reduce computational costs while ensuring control performance.