Abstract:Ureteroscopy with laser lithotripsy has evolved as the most commonly used technique for the treatment of kidney stones. Automated segmentation of kidney stones and the laser fiber is an essential initial step to performing any automated quantitative analysis, particularly stone-size estimation, that can be used by the surgeon to decide if the stone requires further fragmentation. However, factors such as turbid fluid inside the cavity, specularities, motion blur due to kidney movements and camera motion, bleeding, and stone debris impact the quality of vision within the kidney, leading to extended operative times. To the best of our knowledge, this is the first attempt made towards multi-class segmentation in ureteroscopy and laser lithotripsy data. We propose an end-to-end convolution neural network (CNN) based learning framework for the segmentation of stones and laser fiber. The proposed approach utilizes two sub-networks: (I) HybResUNet, a hybrid version of residual U-Net, that uses residual connections in the encoder path of the U-Net to improve semantic predictions, and (II) a DVFNet that generates deformation vector field (DVF) predictions by leveraging motion differences between the adjacent video frames which is then used to prune the prediction maps. We also present ablation studies that combine different dilated convolutions, recurrent and residual connections, atrous spatial pyramid pooling, and attention gate models. Further, we propose a compound loss function that significantly boosts the segmentation performance in our data. We have also provided an ablation study to determine the optimal data augmentation strategy for our dataset. Our qualitative and quantitative results illustrate that our proposed method outperforms state-of-the-art methods such as UNet and DeepLabv3+ showing a DSC improvement of 4.15% and 13.34%, respectively, in our in vivo test dataset. We further show that our proposed model outperforms state-of-the-art methods on an unseen out-of-sample clinical dataset with a DSC improvement of 9.61%, 11%, and 5.24% over UNet, HybResUNet, and DeepLabv3+, respectively in the case of the stone class and an improvement of 31.79%, 22.15%, and 10.42% over UNet, HybResUNet, and DeepLabv3+, respectively, in case of the laser class.

Intelligent Inspection Guidance of Urethral Endoscopy Based on SLAM with Blood Vessel Attentional Features

An Image-guided Endoscope System for the Ureter Detection

Zero-Shot Kidney Stone Segmentation Based on Segmentation Anything Model for Robotic-Assisted Endoscope Navigation.

Laparoscopic Image-Guided System Based on Multispectral Imaging for the Ureter Detection

Real-Time Detection of Ureteral Orifice in Urinary Endoscopy Videos Based on Deep Learning.

A Real-Time System Using Deep Learning to Detect and Track Ureteral Orifices During Urinary Endoscopy.

An Augmented Reality Endoscope System for Ureter Position Detection

Automatic Segmentation of Coronary Lumen and External Elastic Membrane in Intravascular Ultrasound Images Using 8-Layer U-Net

Visual-electromagnetic system: A novel fusion-based monocular localization, reconstruction, and measurement for flexible ureteroscopy

Unsupervised Neural Network-Based Image Stitching Method for Bladder Endoscopy

Semantic SLAM Based on Deep Learning in Endocavity Environment

Spatial Attention Lesion Detection On Automated Breast Ultrasound

A Novel Endoscope System for Position Detection and Depth Estimation of the Ureter

An Automatic Pipeline for Segmentation and Quantification of Intravascular Ultrasound Images

Autonomous Robotic Screening of Tubular Structures based only on Real-Time Ultrasound Imaging Feedback

Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy

Augmented Reality Surgical Navigation System Integrated with Deep Learning

Robotically Surgical Vessel Localization Using Robust Hybrid Video Motion Magnification

Spatial Multi-Scale Attention U-improved Network for Blood Vessel Segmentation

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging

DopUS-Net: Quality-Aware Robotic Ultrasound Imaging based on Doppler Signal