[REPORTS OF THE PHARMACOPOEIAL COMMISSION. 1964, NO. 2].

F. Freycon,M. Freycon

1964-03-27

Pharmaceutisch Weekblad

Abstract:

What problem does this paper attempt to address?

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

Zhen-Liang Ni,Gui-Bin Bian,Xiao-Hu Zhou,Zeng-Guang Hou,Xiao-Liang Xie,Chen Wang,Yan-Jie Zhou,Rui-Qi Li,Zhen Li

DOI: https://doi.org/10.48550/arXiv.1909.10360

2019-10-02

Abstract:Semantic segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, accurate segmentation of cataract surgical instruments is still a challenge due to specular reflection and class imbalance issues. In this paper, an attention-guided network is proposed to segment the cataract surgical instrument. A new attention module is designed to learn discriminative features and address the specular reflection issue. It captures global context and encodes semantic dependencies to emphasize key semantic features, boosting the feature representation. This attention module has very few parameters, which helps to save memory. Thus, it can be flexibly plugged into other networks. Besides, a hybrid loss is introduced to train our network for addressing the class imbalance issue, which merges cross entropy and logarithms of Dice loss. A new dataset named Cata7 is constructed to evaluate our network. To the best of our knowledge, this is the first cataract surgical instrument dataset for semantic segmentation. Based on this dataset, RAUNet achieves state-of-the-art performance 97.71% mean Dice and 95.62% mean IOU.

Computer Vision and Pattern Recognition
Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Zhen-Liang Ni,Gui-Bin Bian,Zeng-Guang Hou,Xiao-Hu Zhou,Xiao-Liang Xie,Zhen Li

DOI: https://doi.org/10.48550/arXiv.1910.11109

2020-09-13

Abstract:The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion block, and transposed convolution. Depthwise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global contexts and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample feature maps for acquiring refined edges. LWANet can segment surgical instruments in real-time while takes little computational costs. Based on 960*544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the-art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with a 4.10% increase on mean IOU.

Computer Vision and Pattern Recognition,Robotics
Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery

Daniil Pakhomov,Nassir Navab

DOI: https://doi.org/10.48550/arXiv.2007.04449

2020-07-09

Abstract:Segmentation of surgical instruments is an important problem in robot-assisted surgery: it is a crucial step towards full instrument pose estimation and is directly used for masking of augmented reality overlays during surgical procedures. Most applications rely on accurate real-time segmentation of high-resolution surgical images. While previous research focused primarily on methods that deliver high accuracy segmentation masks, majority of them can not be used for real-time applications due to their computational cost. In this work, we design a light-weight and highly-efficient deep residual architecture which is tuned to perform real-time inference of high-resolution images. To account for reduced accuracy of the discovered light-weight deep residual network and avoid adding any additional computational burden, we perform a differentiable search over dilation rates for residual units of our network. We test our discovered architecture on the EndoVis 2017 Robotic Instruments dataset and verify that our model is the state-of-the-art in terms of speed and accuracy tradeoff with a speed of up to 125 FPS on high resolution images.

Computer Vision and Pattern Recognition,Image and Video Processing
Real-time instance segmentation of surgical instruments using attention and multi-scale feature fusion

Juan Carlos Ángeles Cerón,Gilberto Ochoa Ruiz,Leonardo Chang,Sharib Ali

DOI: https://doi.org/10.1016/j.media.2022.102569

Abstract:Precise instrument segmentation aids surgeons to navigate the body more easily and increases patient safety. While accurate tracking of surgical instruments in real-time plays a crucial role in minimally invasive computer-assisted surgeries, it is a challenging task to achieve, mainly due to: (1) a complex surgical environment, and (2) model design trade-off in terms of both optimal accuracy and speed. Deep learning gives us the opportunity to learn complex environment from large surgery scene environments and placements of these instruments in real world scenarios. The Robust Medical Instrument Segmentation 2019 challenge (ROBUST-MIS) provides more than 10,000 frames with surgical tools in different clinical settings. In this paper, we propose a light-weight single stage instance segmentation model complemented with a convolutional block attention module for achieving both faster and accurate inference. We further improve accuracy through data augmentation and optimal anchor localization strategies. To our knowledge, this is the first work that explicitly focuses on both real-time performance and improved accuracy. Our approach out-performed top team performances in the most recent edition of ROBUST-MIS challenge with over 44% improvement on area-based multi-instance dice metric MI_DSC and 39% on distance-based multi-instance normalized surface dice MI_NSD. We also demonstrate real-time performance (>60 frames-per-second) with different but competitive variants of our final approach.
A parallel network utilizing local features and global representations for segmentation of surgical instruments

Xinan Sun,Yuelin Zou,Shuxin Wang,He Su,Bo Guan

DOI: https://doi.org/10.1007/s11548-022-02687-z

Abstract:Purpose: Automatic image segmentation of surgical instruments is a fundamental task in robot-assisted minimally invasive surgery, which greatly improves the context awareness of surgeons during the operation. A novel method based on Mask R-CNN is proposed in this paper to realize accurate instance segmentation of surgical instruments. Methods: A novel feature extraction backbone is built, which could extract both local features through the convolutional neural network branch and global representations through the Swin-Transformer branch. Moreover, skip fusions are applied in the backbone to fuse both features and improve the generalization ability of the network. Results: The proposed method is evaluated on the dataset of MICCAI 2017 EndoVis Challenge with three segmentation tasks and shows state-of-the-art performance with an mIoU of 0.5873 in type segmentation and 0.7408 in part segmentation. Furthermore, the results of ablation studies prove that the proposed novel backbone contributes to at least 17% improvement in mIoU. Conclusion: The promising results demonstrate that our method can effectively extract global representations as well as local features in the segmentation of surgical instruments and improve the accuracy of segmentation. With the proposed novel backbone, the network can segment the contours of surgical instruments' end tips more precisely. This method can provide more accurate data for localization and pose estimation of surgical instruments, and make a further contribution to the automation of robot-assisted minimally invasive surgery.
PaI-Net: A Modified U-Net of Reducing Semantic Gap for Surgical Instrument Segmentation

Xiaoyan Wang,Luyao Wang,Xingyu Zhong,Cong Bai,Xiaojie Huang,Ruiyi Zhao,Ming Xia

DOI: https://doi.org/10.1049/ipr2.12283

IF: 2.3

2021-01-01

IET Image Processing

Abstract:Tracking the instruments in a surgical scene is an essential task in minimally invasive surgery. However, due to the unpredictability of scenes, automatically segmenting the instruments is very challenging. In this paper, a novel method named parallel inception network (PaI-Net) is proposed, in which an attention parallel module (APM) and an output fusion module (OFM) are integrated with U-Net to improve the segmentation ability. Specially, APM utilizes multi-scale convolution kernels and global average pooling operations to extract semantic information and global context information of different scales, while OFM combines the feature maps of the decoder part to aggregate the abundant boundary information of shallow layers and the rich semantic information of deep layers together, which achieve a significant improvement in generating segmentation masks. Finally, the evaluation of proposed method on robotic instruments segmentation task from Medical Image Computing and Computer Assisted Intervention Society (MICCAI) and retinal image segmentation task from International Symposium on Biomedical Imaging (ISBI) show that our model has achieved advanced performance on multi-scale semantic segmentation and is superior to the current state-of-the-art models.
Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery

Hongqiu Wang,Lei Zhu,Guang Yang,Yike Guo,Shichen Zhang,Bo Xu,Yueming Jin

DOI: https://doi.org/10.48550/arXiv.2308.09475

2023-08-18

Abstract:Robot-assisted surgery has made significant progress, with instrument segmentation being a critical factor in surgical intervention quality. It serves as the building block to facilitate surgical robot navigation and surgical education for the next generation of operating intelligence. Although existing methods have achieved accurate instrument segmentation results, they simultaneously generate segmentation masks for all instruments, without the capability to specify a target object and allow an interactive experience. This work explores a new task of Referring Surgical Video Instrument Segmentation (RSVIS), which aims to automatically identify and segment the corresponding surgical instruments based on the given language expression. To achieve this, we devise a novel Video-Instrument Synergistic Network (VIS-Net) to learn both video-level and instrument-level knowledge to boost performance, while previous work only used video-level information. Meanwhile, we design a Graph-based Relation-aware Module (GRM) to model the correlation between multi-modal information (i.e., textual description and video frame) to facilitate the extraction of instrument-level information. We are also the first to produce two RSVIS datasets to promote related research. Our method is verified on these datasets, and experimental results exhibit that the VIS-Net can significantly outperform existing state-of-the-art referring segmentation methods. Our code and our datasets will be released upon the publication of this work.

Computer Vision and Pattern Recognition
[Pulmonary interstitial fibrosis as a systemic manifestation of active chronic hepatitis].

S. Echevarría Vierna,J. Riancho,M. Fariñas,R. Ondiviela,J. González Macías

1985-10-19

Abstract:
Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

Liping Sun,Xiong Chen

DOI: https://doi.org/10.1109/access.2024.3476622

IF: 3.9

2024-01-01

IEEE Access

Abstract:The accurate segmentation of surgical instruments in endoscopic robotic surgery is a critical challenge due to the intricate and dynamic nature of the surgical environment. Existing segmentation techniques predominantly focus on binary classification, which often falls short in complex scenarios where multiple instruments need to be precisely identified and differentiated. This limitation significantly hampers the effectiveness of computer-assisted surgical systems, where real-time and accurate multi-class segmentation is paramount. In this study, we address these challenges by introducing a novel segmentation framework that leverages pixel-wise contrastive learning to enhance multi-class instrument segmentation. Our approach integrates the strengths of HRNet and DeepLabV3 as backbone networks to effectively capture and embed high-resolution features from endoscopic images. These feature embeddings are stored in dynamically updated dataset-wide queues, enabling the model to mitigate the class imbalance by incorporating representative samples from less prevalent classes during training. This is achieved by combining an enhanced contrastive loss function with pixel-wise cross-entropy loss, which together facilitate robust multi-class differentiation. We validate our approach using the challenging EndoVis 2017 and EndoVis 2018 datasets, where it demonstrated superior performance compared to existing methods. On the EndoVis 2017 test dataset, our HRNet-based method achieved an average mIOU of 0.682 across 10 test subsets, surpassing the official benchmark of 0.542 by a significant margin. Similarly, on the EndoVis 2018 test dataset, our method attained an average mIOU of 0.599, exceeding the official best score of 0.579. These results underscore the efficacy of our approach in significantly improving segmentation accuracy. The enhanced segmentation capability not only advances the state of the art in multi-class instrument segmentation but also has practical implications for improving surgical safety and efficiency in real-world robotic surgery.
Dual-stage semantic segmentation of endoscopic surgical instruments

Wenxin Chen,Kaifeng Wang,Xinya Song,Dongsheng Xie,Xue Li,Mobarakol Islam,Changsheng Li,Xingguang Duan

DOI: https://doi.org/10.1002/mp.17397

2024-09-10

Abstract:Background: Endoscopic instrument segmentation is essential for ensuring the safety of robotic-assisted spinal endoscopic surgeries. However, due to the narrow operative region, intricate surrounding tissues, and limited visibility, achieving instrument segmentation within the endoscopic view remains challenging. Purpose: This work aims to devise a method to segment surgical instruments in endoscopic video. By designing an endoscopic image classification model, features of frames before and after the video are extracted to achieve continuous and precise segmentation of instruments in endoscopic videos. Methods: Deep learning techniques serve as the algorithmic core for constructing the convolutional neural network proposed in this study. The method comprises dual stages: image classification and instrument segmentation. MobileViT is employed for image classification, enabling the extraction of key features of different instruments and generating classification results. DeepLabv3+ is utilized for instrument segmentation. By training on distinct instruments separately, corresponding model parameters are obtained. Lastly, a flag caching mechanism along with a blur detection module is designed to effectively utilize the image features in consecutive frames. By incorporating specific parameters into the segmentation model, better segmentation of surgical instruments can be achieved in endoscopic videos. Results: The classification and segmentation models are evaluated on an endoscopic image dataset. In the dataset used for instrument segmentation, the training set consists of 7456 images, the validation set consists of 829 images, and the test set consists of 921 images. In the dataset used for image classification, the training set consists of 2400 images and the validation set consists of 600 images. The image classification model achieves an accuracy of 70% on the validation set. For the segmentation model, experiments are conducted on two common surgical instruments, and the mean Intersection over Union (mIoU) exceeds 98%. Furthermore, the proposed video segmentation method is tested using videos collected during surgeries, validating the effectiveness of the flag caching mechanism and blur detection module. Conclusions: Experimental results on the dataset demonstrate that the dual-stage video processing method excels in performing instrument segmentation tasks under endoscopic conditions. This advancement is significant for enhancing the intelligence level of robotic-assisted spinal endoscopic surgeries.
Semantic Segmentation of Surgical Instruments Based on Enhanced Multi-scale Receptive Field

Yu Dong,Hongbo Wang,Jingjing Luo,Zhiping Lai,Fuhao Wang,Jiawei Wang

DOI: https://doi.org/10.1088/1742-6596/2003/1/012006

2021-01-01

Journal of Physics Conference Series

Abstract:With the rapid development of robot assisted surgery, the segmentation of surgical instruments becomes more and more important. However, compared with the natural scene segmentation, surgical instrument segmentation is more difficult. To solve this problem, we improve a high and low resolution fusion module, which aims to extract detail information and context information from the fusion feature map of high and low resolution. Then, in the last layer of the encoder, we propose the Enhanced Multi-scale Receptive Field module to generate more available receptive fields. Our method is validated on 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset, and the result is better than the other methods. The extended experiment is carried out on the dataset of our surgical soft robot which has a content implementation.
SINet: A hybrid deep CNN model for real-time detection and segmentation of surgical instruments

Zhenzhong Liu,Yifan Zhou,Laiwang Zheng,Guobin Zhang

DOI: https://doi.org/10.1016/j.bspc.2023.105670

IF: 5.1

2023-11-05

Biomedical Signal Processing and Control

Abstract:Objective Detection and segmentation of surgical instruments is an indispensable technology in robot-assisted surgery that enables doctors to obtain more comprehensive visual information and further improve the safety of surgery. However, the results of a detection are more easily interfered by environmental factors, such as instrument shaking, incomplete displays and insufficient light. To overcome those issues, we designed a hybrid deep-CNN model (SINet) for real-time surgical instrument detection and segmentation. Methods The framework employs YOLOv5 as the object detection model and introduces a GAM attention mechanism to improve its feature extraction abilities. During training, the SiLU activation function is adopted to avoid gradient explosions and unstable training situations. Specifically, the vector angle relationship between the ground truth boxes and the prediction boxes was applied in the SIoU loss function to reduce the degree of freedom of the regression and accelerate the network convergence. Finally, a semantic segmentation head is used to implement detections of the surgical instruments by paralleling the detection and segmentation. Results The proposed method is evaluated on the m2cai16-tool-locations public dataset and achieved a significant 97.9% mean average precision (mAP), 133 frames per second (FPS), 85.7% mean intersection over union (MIoU) and 86.6% Dice. Experiment based on simulated surgery platform also shows satisfactory detection performance. Conclusion Experimental results demonstrated that the SINet can effectively detect the pose of surgical instruments and achieves a better performance than most of the current algorithms. The method has the potential to help perform a series of surgical operations efficiently and safely.

engineering, biomedical
Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Daniil Pakhomov,Vittal Premachandran,Max Allan,Mahdi Azizian,Nassir Navab

DOI: https://doi.org/10.48550/arXiv.1703.08580

2017-03-25

Abstract:Detection, tracking, and pose estimation of surgical instruments are crucial tasks for computer assistance during minimally invasive robotic surgery. In the majority of cases, the first step is the automatic segmentation of surgical tools. Prior work has focused on binary segmentation, where the objective is to label every pixel in an image as tool or background. We improve upon previous work in two major ways. First, we leverage recent techniques such as deep residual learning and dilated convolutions to advance binary-segmentation performance. Second, we extend the approach to multi-class segmentation, which lets us segment different parts of the tool, in addition to background. We demonstrate the performance of this method on the MICCAI Endoscopic Vision Challenge Robotic Instruments dataset.

Computer Vision and Pattern Recognition
Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network

Yu-Wen Luo,Hai-Yong Chen,Zhen Li,Wei-Peng Liu,Ke Wang,Li Zhang,Pan Fu,Wen-Qian Yue,Gui-Bin Bian

DOI: https://doi.org/10.1016/j.compbiomed.2022.106531

IF: 7.7

2023-01-05

Computers in Biology and Medicine

Abstract:Surgical scene segmentation provides critical information for guidance in micro-neurosurgery. Segmentation of instruments and critical tissues contributes further to robot assisted surgery and surgical evaluation. However, due to the lack of relevant scene segmentation dataset, scale variation and local similarity, micro-neurosurgical segmentation faces many challenges. To address these issues, a high correlative non-local network (HCNNet), is proposed to aggregate multi-scale feature by optimized non-local mechanism. HCNNet adopts two-branch design to generate features of different scale efficiently, while the two branches share common weights in shallow layers. Several short-term dense concatenate (STDC) modules are combined as the backbone to capture both semantic and spatial information. Besides, a high correlative non-local module (HCNM) is designed to guide the upsampling process of the high-level feature by modeling global context generated from the low-level feature. It filters out confused pixels of different classes in the non-local correlation map. Meanwhile, a large segmentation dataset named NeuroSeg is constructed, which contains 15 types of instruments and 3 types of tissues that appear in meningioma resection surgery. The proposed HCNNet achieves the state-of-the-art performance on NeuroSeg, it reaches an inference speed of 54.85 FPS with the highest accuracy of 59.62% mIoU, 74.7% Dice, 70.55% mAcc and 87.12% aAcc.

engineering, biomedical,computer science, interdisciplinary applications,mathematical & computational biology,biology
Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions

S M Kamrul Hasan,Richard A Simon,Cristian A Linte

DOI: https://doi.org/10.1117/1.JMI.10.4.045002

Abstract:Purpose: Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). Approach: With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. Results: We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Conclusions: Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.
Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning

Mobarakol Islam,Daniel A. Atputharuban,Ravikiran Ramesh,Hongliang Ren

DOI: https://doi.org/10.1109/LRA.2019.2900854

2020-09-30

Abstract:Robot-assisted surgery is an emerging technology which has undergone rapid growth with the development of robotics and imaging systems. Innovations in vision, haptics and accurate movements of robot arms have enabled surgeons to perform precise minimally invasive surgeries. Real-time semantic segmentation of the robotic instruments and tissues is a crucial step in robot-assisted surgery. Accurate and efficient segmentation of the surgical scene not only aids in the identification and tracking of instruments but also provided contextual information about the different tissues and instruments being operated with. For this purpose, we have developed a light-weight cascaded convolutional neural network (CNN) to segment the surgical instruments from high-resolution videos obtained from a commercial robotic system. We propose a multi-resolution feature fusion module (MFF) to fuse the feature maps of different dimensions and channels from the auxiliary and main branch. We also introduce a novel way of combining auxiliary loss and adversarial loss to regularize the segmentation model. Auxiliary loss helps the model to learn low-resolution features, and adversarial loss improves the segmentation prediction by learning higher order structural information. The model also consists of a light-weight spatial pyramid pooling (SPP) unit to aggregate rich contextual information in the intermediate stage. We show that our model surpasses existing algorithms for pixel-wise segmentation of surgical instruments in both prediction accuracy and segmentation time of high-resolution videos.

Computer Vision and Pattern Recognition,Robotics,Image and Video Processing
Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision

Fangbo Qin,Shan Lin,Yangming Li,Randall A. Bly,Kris S. Moe,Blake Hannaford

DOI: https://doi.org/10.1109/LRA.2020.3009073

2020-08-11

Abstract:Accurate and real-time surgical instrument segmentation is important in the endoscopic vision of robot-assisted surgery, and significant challenges are posed by frequent instrument-tissue contacts and continuous change of observation perspective. For these challenging tasks more and more deep neural networks (DNN) models are designed in recent years. We are motivated to propose a general embeddable approach to improve these current DNN segmentation models without increasing the model parameter number. Firstly, observing the limited rotation-invariance performance of DNN, we proposed the Multi-Angle Feature Aggregation (MAFA) method, leveraging active image rotation to gain richer visual cues and make the prediction more robust to instrument orientation changes. Secondly, in the end-to-end training stage, the auxiliary contour supervision is utilized to guide the model to learn the boundary awareness, so that the contour shape of segmentation mask is more precise. The proposed method is validated with ablation experiments on the novel Sinus-Surgery datasets collected from surgeons' operations, and is compared to the existing methods on a public dataset collected with a da Vinci Xi Robot.

Computer Vision and Pattern Recognition
Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion

Daochang Liu,Yuhui Wei,Tingting Jiang,Yizhou Wang,Rulin Miao,Fei Shan,Ziyu Li

DOI: https://doi.org/10.1007/978-3-030-59716-0_63

2020-01-01

Abstract: Surgical instrument segmentation is a key component in developing context-aware operating rooms. Existing works on this task heavily rely on the supervision of a large amount of labeled data, which involve laborious and expensive human efforts. In contrast, a more affordable unsupervised approach is developed in this paper. To train our model, we first generate anchors as pseudo labels for instruments and background tissues respectively by fusing coarse handcrafted cues. Then a semantic diffusion loss is proposed to resolve the ambiguity in the generated anchors via the feature correlation between adjacent video frames. In the experiments on the binary instrument segmentation task of the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset, the proposed method achieves 0.71 IoU and 0.81 Dice score without using a single manual annotation, which is promising to show the potential of unsupervised learning for surgical tool segmentation.
Multiscale Matters for Part Segmentation of Instruments in Robotic Surgery.

Wenhao He,Haitao Song,Yue Guo,Guibin Bian,Yuejie Sun,Xiaowei Zhou,Xiaonan Wang

DOI: https://doi.org/10.1049/iet-ipr.2020.0320

IF: 2.3

2020-01-01

IET Image Processing

Abstract:A challenging aspect of instrument segmentation in robotic surgery is to distinguish different parts of the same instrument. Parts with similar textures are common in a practical instrument and are difficult to distinguish. In this work, the authors introduce an end-to-end recurrent model that comprises a multiscale semantic segmentation network and a refinement model. Specifically, the semantic segmentation network uniformly transforms the input images in multiple scales into a semantic mask, and the refinement model is a single-scale net recurrently optimising the above semantic mask. Through extensive experiments, the authors validate that the models with multiscale inputs perform better than those to fuse encoded feature maps and ones with spatial attention. Furthermore, the authors verify the effectiveness of the proposed model with state-of-the-art performances on several robotic instrument datasets derived from MICCAI Endoscopic Vision Challenges.
Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Wenhan Lin,Jinze Shi,Honghai Ma,Luming Wang,Jian Hu,Chunlin Zhou

DOI: https://doi.org/10.1109/IECON51785.2023.10312123

2023-01-01

Abstract:Robot-assist laparoscopic surgery has been widely acknowledged in the minimally invasive surgery for its capability of operating within limited space and its robustness. Particularly, the vision-guided camera-holder system is suggested worldwide for it could track the instruments autonomously without additional equipment and its low cost, in which computer vision contributes a lot to the recognition and localization of the surgical instruments. This paper proposes vision-based real-time tracking of three basic surgical instruments in laparoscopic surgery. Two algorithms in computer vision are compared for instruments detection - Template Matching and Deep Learning, and the real-time tracking is completed based on the detection. Optimizations are made for both of algorithms. For Template Matching, three methods to enhance efficiency are applied. For Deep Learning, different loss functions and Attention Mechanisms are introduced and compared. Models are evaluated by the metrics of Precision, Recall and Detecting Time. The results reveal that despite the high precision of 0.83, the detection time of Template Matching reaches above 5000 ms, which is too slow to meet the real-time requirements in the surgery, while YOLOv5 models with CIoU and SENet performs best overall with the precision of 0.98 and the detection time of around 20 ms.

[REPORTS OF THE PHARMACOPOEIAL COMMISSION. 1964, NO. 2].

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery

Real-time instance segmentation of surgical instruments using attention and multi-scale feature fusion

A parallel network utilizing local features and global representations for segmentation of surgical instruments

PaI-Net: A Modified U-Net of Reducing Semantic Gap for Surgical Instrument Segmentation

Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery

[Pulmonary interstitial fibrosis as a systemic manifestation of active chronic hepatitis].

Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

Dual-stage semantic segmentation of endoscopic surgical instruments

Semantic Segmentation of Surgical Instruments Based on Enhanced Multi-scale Receptive Field

SINet: A hybrid deep CNN model for real-time detection and segmentation of surgical instruments

Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions

Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning

Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision

Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion

Multiscale Matters for Part Segmentation of Instruments in Robotic Surgery.

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery