Abstract:Abstract Steam injection is a popular technique to enhance oil recovery in mature oil fields. However, the conventional approach of using a constant steam rate over an extended period can lead to sub-optimal performance due to the complex nature of the problem and reservoir heterogeneity. To address this issue, the Markov decision process can be employed to formulate the problem for reinforcement learning (RL) applications. The RL agent is trained to optimize the steam injection rate by interacting with a reservoir simulation model and receives rewards for each action. The agent’s policy and value functions are updated through continuous interaction with the environment until convergence is achieved, leading to a more efficient steam injection strategy for enhancing oil recovery. In this study, an actor-critic RL architecture was employed to train the agent to find the optimal strategy (i.e., policy). The environment was represented by a reservoir simulation model, and the agent’s actions were based on the observed state. The policy function gave a probability distribution of the actions that the agent could take, while the value function determined the expected yield for an agent starting from a given state. The agent interacted with the environment for several episodes until convergence was achieved. The improvement in net present value (NPV) achieved by the agent was a significant indication of the effectiveness of the RL-based approach. The NPV reflects the economic benefits of the optimized steam injection strategy. The agent was able to achieve this improvement by finding the optimal policies. One of the key advantages of the optimal policy was the decrease in total field heat losses. This is a critical factor in the efficiency of the steam injection process. Heat loss can reduce the efficiency of the process and lead to lower oil recovery rates. By minimizing heat loss, the agent was able to optimize the steam injection process and increase oil recovery rates. The optimal policy had four regions characterized by slight changes in a stable injection rate to increase the average reservoir pressure, increasing the injection rate to a maximum value, steeply decreasing the injection rate, and slightly changing the injection rate to maintain the average reservoir temperature. These regions reflect the different phases of the steam injection process and demonstrate the complexity of the problem. Overall, the results of this study demonstrate the effectiveness of RL in optimizing steam injection in mature oil fields. The use of RL can help address the complexity of the problem and improve the efficiency of the oil recovery process. This study provides a framework for future research in this area and highlights the potential of RL for addressing other complex problems in the energy industry.

Reinforcement Learning Based Decision Making of Operational Indices in Process Industry Under Changing Environment

Accelerating Reinforcement Learning with Local Data Enhancement for Process Control

Human operator decision support for highly transient industrial processes: a reinforcement learning approach

Data-Based Multiobjective Plant-Wide Performance Optimization of Industrial Processes Under Dynamic Environments

Reinforcement Learning in Process Industries: Review and Perspective

Multimodal Data-Driven Reinforcement Learning for Operational Decision-Making in Industrial Processes

Development of Parametric Reinforcement Learning for different operation preferences

Model-free Adaptive Optimal Control of Episodic Fixed-horizon Manufacturing Processes Using Reinforcement Learning

Accelerating reinforcement learning with case-based model-assisted experience augmentation for process control

Towards Solving Industrial Sequential Decision-making Tasks under Near-predictable Dynamics via Reinforcement Learning: an Implicit Corrective Value Estimation Approach

Nonzero-Sum Game Reinforcement Learning for Performance Optimization in Large-Scale Industrial Processes

Reinforcement learning algorithm for non-stationary environments

Safe reinforcement learning method integrating process knowledge for real-time scheduling of gas supply network

Distributional Reinforcement Learning for Scheduling of Chemical Production Processes

Online Operational Decision-making for Integrated Electric-Gas Systems with Safe Reinforcement Learning

Deep reinforcement learning for process design: Review and perspective

A review On reinforcement learning: Introduction and applications in industrial process control

Actor-critic reinforcement learning leads decision-making in energy systems optimization—steam injection optimization

Evolutionary-assisted reinforcement learning for reservoir real-time production optimization under uncertainty

Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis

Intelligent Scheduling of Discrete Automated Production Line Via Deep Reinforcement Learning