Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning

Yafei Ou,Mahdi Tavakoli

2024-11-22

Abstract:The irrigation-suction process is a common procedure to rinse and clean up the surgical field in minimally invasive surgery (MIS). In this process, surgeons first irrigate liquid, typically saline, into the surgical scene for rinsing and diluting the contaminant, and then suction the liquid out of the surgical field. While recent advances have shown promising results in the application of reinforcement learning (RL) for automating surgical subtasks, fewer studies have explored the automation of fluid-related tasks. In this work, we explore the automation of both steps in the irrigation-suction procedure and train two vision-based RL agents to complete irrigation and suction autonomously. To achieve this, a platform is developed for creating simulated surgical robot learning environments and for training agents, and two simulated learning environments are built for irrigation and suction with visually plausible fluid rendering capabilities. With techniques such as domain randomization (DR) and carefully designed reward functions, two agents are trained in the simulator and transferred to the real world. Individual evaluations of both agents show satisfactory real-world results. With an initial amount of around 5 grams of contaminants, the irrigation agent ultimately achieved an average of 2.21 grams remaining after a manual suction. As a comparison, fully manual operation by a human results in 1.90 grams remaining. The suction agent achieved 2.64 and 2.24 grams of liquid remaining across two trial groups with more than 20 and 30 grams of initial liquid in the container. Fully autonomous irrigation-suction trials reduce the contaminant in the container from around 5 grams to an average of 2.42 grams, although yielding a higher total weight remaining (4.40) due to residual liquid not suctioned. Further information about the project is available at <a class="link-external link-https" href="https://tbs-ualberta.github.io/CRESSim/" rel="external noopener nofollow">this https URL</a>.

Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the irrigation and suction processes in automated minimally invasive surgery (MIS) to reduce the workload of surgeons. Specifically:

1. **Problem Background**:
- In minimally invasive surgery, the irrigation - suction process is a common procedure for irrigating and cleaning the surgical area. This process consists of two steps: first, the surgeon injects a liquid (usually normal saline) into the surgical scene to irrigate and dilute contaminants; then, a suction device is used to suck the liquid out of the surgical area.
- Although significant progress has been made in recent years in automating surgical subtasks using reinforcement learning (RL), there has been less research on the automation of tasks involving fluids, such as irrigation and suction.

2. **Research Objectives**:
- This research aims to explore the two steps of the automated irrigation - suction process and train vision - based reinforcement learning agents to complete these two steps autonomously.
- By developing a simulation environment and training agents, ultimately achieve sim - to - real transfer from the simulation environment to the real world, so as to apply these automation techniques in actual surgeries.

3. **Specific Problems**:
- How to simulate complex fluid dynamics and the corresponding interactions in order to accurately reproduce the irrigation and suction processes in the simulation environment.
- How to design a vision - based strategy so that the agent can learn directly from the raw RGB images captured by the endoscope camera without manual feature extraction.
- How to ensure that the trained agent can effectively perform tasks in the real world, especially maintain robustness under different surgical conditions.

4. **Solutions**:
- A new surgical robot learning platform, CRESSim - ML, has been developed, integrating robotic learning capabilities and visually realistic fluid simulation and rendering.
- Two simulation learning environments have been designed for the irrigation and suction tasks respectively, and these environments can simulate real - world settings.
- Two vision - based agents have been trained using domain randomization (DR), carefully designed reward functions, curriculum learning (CL) and imitation learning (IL), enabling them to control the robot to complete tasks in the joint space.
- The transfer effects of these agents from the simulation environment to the real world have been proven through experiments, and their performance has been quantitatively analyzed.

In summary, the main contribution of this paper is that it is the first to explore the automation of the irrigation and suction processes and proposes a vision - based reinforcement learning method, enabling surgical robots to autonomously perform tasks based on RGB image observations without additional feature extraction.