Abstract:Abstract During drilling operations, maintaining a desired downhole pressure between pressure margins is crucial to avoid damage to the formation and the well. The process is highly nonlinear, changing with depth, and every section in every well is different. Standard solutions with PID controllers are widely accepted for this purpose, although methods such as deep reinforcement learning (DRL) could be investigated as an alternative approach. A smooth update deep Q learning algorithm is used to train an agent embedded in a managed pressure drilling system. The aim is to control downhole pressure during pipe connections by use of a topside choke valve with nonlinear characteristics. The agent is trained on previously gathered data, from situations featuring step changes in pressure setpoint and changing mud flows, all at various well depths. After training, the agent is tasked with controlling BHP during connection, herein demonstrated by use of a numerically simulated low-order hydraulics model. Through episodic training, it becomes clear that the agent purely through interaction with the environment, and without any prerequisite knowledge of system dynamics and reward design, converges to an optimal control policy. The trained agent is then tested on pipe connections with well depths in the lower and upper bounds of the training data. The pipe connection scenario presents changes in operating conditions in terms of changing mud flows with changing conditions like increased frictional pressure losses due to increased depth. Still, the results presented show the agent's ability to track a pressure setpoint at various depths in the changing conditions present during connection, while seamlessly incorporating controller constraints. There are several advantages associated with this approach, among them eliminating the need for development of a complex dynamic model for the process. Also, the approach is applicable to both linear and nonlinear systems, deterministic and stochastic systems, and lower- and higher-level decision-making. These methods could possibly also be applied to other key challenges in drilling such as ROP optimization or autonomous directional drilling.

Deep reinforcement learning in dynamic positioning control: by rewarding small response of riser angles

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Deep Reinforcement Learning Based Controller for Active Heave Compensation

Deep Reinforcement Learning Applied to Managed Pressure Drilling

Asynchronous Localization for Underwater Acoustic Sensor Networks: A Continuous Control Deep Reinforcement Learning Approach

Algorithms for dynamic control of a deep-sea mining vehicle based on deep reinforcement learning

Single degree of freedom control based on deep reinforcement learning for underwater unmanned vehicle

DRIFT: Deep Reinforcement Learning for Intelligent Floating Platforms Trajectories

AI on the Water: Applying DRL to Autonomous Vessel Navigation

Harnessing traditional controllers for fast-track training of deep reinforcement learning control strategies

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method

Navigating the Ocean with DRL: Path following for marine vessels

Multi-objective optimization framework for deepwater riser jetting installation parameters using deep reinforcement learning

A reinforcement learning control approach for underwater manipulation under position and torque constraints

Barrier Lyapunov function-based adaptive optimized control for full-state and input-constrained dynamic positioning of marine vessels with simulation and model-scale tests

Research and Design of an Autonomous Underwater Vehicle Path Planning Method Based on Deep Reinforcement Learning

Deep Reinforcement Learning for Motion Control Algorithms in Robotics

Safe reward‐based deep reinforcement learning control for an electro‐hydraulic servo system

Deep Reinforcement Learning for Continuous Docking Control of Autonomous Underwater Vehicles: A Benchmarking Study

Intelligent ship anti-rolling control system based on a deep deterministic policy gradient algorithm and the Magnus effect

A Deep Reinforcement Learning-based Decentralized Hierarchical Motion Control Strategy for Multiple Amphibious Spherical Robot Systems with Tilting Thrusters