Abstract:Purpose: Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). Approach: With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. Results: We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Conclusions: Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.

DisguisOR: holistic face anonymization for the operating room

DisguisOR: Holistic Face Anonymization for the Operating Room

3D Holistic OR Anonymization

FaceOff: Anonymizing Videos in the Operating Rooms

Privacy-proof Live Surgery Streaming

Privacy-proof Live Surgery Streaming: Development and Validation of a Low-cost, Real-time Robotic Surgery Anonymization Algorithm

A Robotic 3D Perception System for Operating Room Environment Awareness

Toward Intraoperative Visual Intelligence: Real-Time Surgical Instrument Segmentation for Enhanced Surgical Monitoring

Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videos

Monocular pose estimation of articulated surgical instruments in open surgery

Disguise without Disruption: Utility-Preserving Face De-Identification

Practical Digital Disguises: Leveraging Face Swaps to Protect Patient Privacy

VerA: Versatile Anonymization Applicable to Clinical Facial Photographs

Privacy-Preserving Action Recognition for Smart Hospitals using Low-Resolution Depth Images

Exploring AI-based Anonymization of Industrial Image and Video Data in the Context of Feature Preservation

Ensuring privacy protection in the era of big laparoscopic video data: development and validation of an inside outside discrimination algorithm (IODA)

VISAGE: Video Synthesis using Action Graphs for Surgery

Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera

Learning to Anonymize Faces for Privacy Preserving Action Detection

Automatic Detection of Out-of-body Frames in Surgical Videos for Privacy Protection Using Self-supervised Learning and Minimal Labels

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions