Capsaicin-induced biphasic oxygen uptake in rat muscle: antagonism by capsazepine and ruthenium red provides further evidence for peripheral vanilloid receptor subtypes (VN1/VN2).

C. Griffiths,T. Eldershaw,D. Geraghty,J. Hall,E. Q. Colquhoun

DOI: https://doi.org/10.1016/0024-3205(96)00267-6

1996-06-07

Life Science

Abstract:

What problem does this paper attempt to address?

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Wenhan Lin,Jinze Shi,Honghai Ma,Luming Wang,Jian Hu,Chunlin Zhou

DOI: https://doi.org/10.1109/IECON51785.2023.10312123

2023-01-01

Abstract:Robot-assist laparoscopic surgery has been widely acknowledged in the minimally invasive surgery for its capability of operating within limited space and its robustness. Particularly, the vision-guided camera-holder system is suggested worldwide for it could track the instruments autonomously without additional equipment and its low cost, in which computer vision contributes a lot to the recognition and localization of the surgical instruments. This paper proposes vision-based real-time tracking of three basic surgical instruments in laparoscopic surgery. Two algorithms in computer vision are compared for instruments detection - Template Matching and Deep Learning, and the real-time tracking is completed based on the detection. Optimizations are made for both of algorithms. For Template Matching, three methods to enhance efficiency are applied. For Deep Learning, different loss functions and Attention Mechanisms are introduced and compared. Models are evaluated by the metrics of Precision, Recall and Detecting Time. The results reveal that despite the high precision of 0.83, the detection time of Template Matching reaches above 5000 ms, which is too slow to meet the real-time requirements in the surgery, while YOLOv5 models with CIoU and SENet performs best overall with the precision of 0.98 and the detection time of around 20 ms.
Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Xiaofei Du,Maximilian Allan,Alessio Dore,Sebastien Ourselin,David Hawkes,John D. Kelly,Danail Stoyanov

DOI: https://doi.org/10.1007/s11548-016-1393-4

2016-04-02

International Journal of Computer Assisted Radiology and Surgery

Abstract:PurposeComputer-assisted interventions for enhanced minimally invasive surgery (MIS) require tracking of the surgical instruments. Instrument tracking is a challenging problem in both conventional and robotic-assisted MIS, but vision-based approaches are a promising solution with minimal hardware integration requirements. However, vision-based methods suffer from drift, and in the case of occlusions, shadows and fast motion, they can be subject to complete tracking failure.MethodsIn this paper, we develop a 2D tracker based on a Generalized Hough Transform using SIFT features which can both handle complex environmental changes and recover from tracking failure. We use this to initialize a 3D tracker at each frame which enables us to recover 3D instrument pose over long sequences and even during occlusions.ResultsWe quantitatively validate our method in 2D and 3D with ex vivo data collected from a DVRK controller as well as providing qualitative validation on robotic-assisted in vivo data.ConclusionsWe demonstrate from our extended sequences that our method provides drift-free robust and accurate tracking. Our occlusion-based sequences additionally demonstrate that our method can recover from occlusion-based failure. In both cases, we show an improvement over using 3D tracking alone suggesting that combining 2D and 3D tracking is a promising solution to challenges in surgical instrument tracking.

engineering, biomedical,radiology, nuclear medicine & medical imaging,surgery
Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation

Junjun Pan,Weimin Liu,Pu Ge,Fanghong Li,Weiyun Shi,Liyun Jia,Hong Qin

DOI: https://doi.org/10.1016/j.cmpb.2020.105679

Abstract:Objective: Corneal disease is one of the main causes of blindness for humans globally nowadays, and deep anterior lamellar keratoplasty (DALK) is a widely applied technique for corneal transplantation. However, the position of stitch points highly influences the success rate of such surgery, which would require accurate control and manipulation of surgical instruments. Methods: In this paper, we present a deep learning framework for augmented reality (AR) based surgery navigation to guide the suturing in DALK. It can robustly track the excised corneal contour by semantic segmentation and the reconstruction of occlusion. We propose a novel optical flow inpainting network to recover the missing motion caused by occlusion. The occluded regions are detected by weakly supervised segmentation of surgical instruments and reconstructed by key frame warping along the completed optical flow. Then we introduce two types of loss function to adapt the inpainting network in the optical flow space. Results: Our techniques are tested and evaluated by a number of real surgery videos from Shandong Eye Hospital in China. We compare our approaches with other typical methods in the corneal contour segmentation, optical flow inpainting and occlusion regions reconstruction. The tracking accuracy reachs 99.2% in average and PSNR reaches 25.52 for the reconstruction of the occluded frames. Conclusion: From the experimental evaluations and user study, both the qualitative and quantitative results indicate that our techniques can achieve accurate detection and tracking of corneal contour under complex disturbance in real-time surgical scenes. Our prototype AR navigation system would be highly useful in clinical practice.
DeepPhase: Surgical Phase Recognition in CATARACTS Videos

Odysseas Zisimopoulos,Evangello Flouty,Imanol Luengo,Petros Giataganas,Jean Nehme,Andre Chow,Danail Stoyanov

DOI: https://doi.org/10.48550/arXiv.1807.10565

2018-07-17

Abstract:Automated surgical workflow analysis and understanding can assist surgeons to standardize procedures and enhance post-surgical assessment and indexing, as well as, interventional monitoring. Computer-assisted interventional (CAI) systems based on video can perform workflow estimation through surgical instruments' recognition while linking them to an ontology of procedural phases. In this work, we adopt a deep learning paradigm to detect surgical instruments in cataract surgery videos which in turn feed a surgical phase inference recurrent network that encodes temporal aspects of phase steps within the phase classification. Our models present comparable to state-of-the-art results for surgical tool detection and phase recognition with accuracies of 99 and 78% respectively.

Computer Vision and Pattern Recognition,Machine Learning
Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Bo Guan,Yuelin Zou,Jianchang Zhao,Lizhi Pan,Bo Yi,Jianmin Li

DOI: https://doi.org/10.1016/j.compbiomed.2023.107472

Abstract:Robot-assisted minimally invasive surgery has been broadly employed in complicated operations. However, the multiple surgical instruments may occupy a large amount of visual space in complex operations performed in narrow spaces, which affects the surgeon's judgment on the shape and position of the lesion as well as the course of its adjacent vessels/lacunae. In this paper, a surgical scene reconstruction method is proposed, which involves the tracking and removal of surgical instruments and the dynamic prediction of the obscured region. For tracking and segmentation of instruments, the image sequences are preprocessed by a modified U-Net architecture composed of a pre-trained ResNet101 encoder and a redesigned decoder. Also, the segmentation boundaries of the instrument shafts are extended using image filtering and a real-time index mask algorithm to achieve precise localization of the obscured elements. For predicting the deformation of soft tissues, a soft tissue deformation prediction algorithm is proposed based on dense optical flow gravitational field and entropy increase, which can achieve local dynamic visualization of the surgical scene by integrating image morphological operations. Finally, the preliminary experiments and the pre-clinical evaluation were presented to demonstrate the performance of the proposed method. The results show that the proposed method can provide the surgeon with a clean and comprehensive surgical scene, reconstruct the course of important vessels/lacunae, and avoid inadvertent injuries.
Swept Source Intraoperative OCT Angiography

Ziyi Zhang,Tiepei Zhu,Tongtong Cao,Zhaoyu Gong,Lin Yao,Kaiyuan Liu,Juan Ye,Peng Li

DOI: https://doi.org/10.1142/s1793545821400095

2021-01-01

Abstract:To accurately guide surgical instruments during ophthalmic procedures, some necessary intraoperative depth perception is required, which standard surgical microscopes supply limitedly. Intraoperative optical coherence tomography (iOCT), combining optical coherence tomography (OCT) technology and surgical microscope, enables noninvasive, real-time and high-resolution cross-sectional imaging. Currently, though iOCT enables structural imaging, little research has been done on intraoperative angiography. In this work, we presented a swept-source intraoperative OCT angiography (SS-iOCTA) system based on a standard surgical microscope, which provides both structural and angiographic images. The feasibility of the proposed SS-iOCTA was confirmed through deep anterior lamellar keratoplasty (DALK) of ex vivo porcine eyes and blood perfusion imaging of in vivo rat cortex. High-resolution intraoperative feedback, including sub-surface structure and angiogram of biological tissue, can be visualized simultaneously with the SS-iOCTA system, which expand the surgeon’s capabilities and could be widely used in clinical surgery.
Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer

Puxun Tu,Hongfei Ye,Haochen Shi,Jeff Young,Meng Xie,Peiquan Zhao,Ce Zheng,Xiaoyi Jiang,Xiaojun Chen

DOI: https://doi.org/10.48550/arXiv.2309.05209

2023-11-01

Abstract:Phacoemulsification cataract surgery (PCS) is a routine procedure conducted using a surgical microscope, heavily reliant on the skill of the ophthalmologist. While existing PCS guidance systems extract valuable information from surgical microscopic videos to enhance intraoperative proficiency, they suffer from non-phasespecific guidance, leading to redundant visual information. In this study, our major contribution is the development of a novel phase-specific augmented reality (AR) guidance system, which offers tailored AR information corresponding to the recognized surgical phase. Leveraging the inherent quasi-standardized nature of PCS procedures, we propose a two-stage surgical microscopic video recognition network. In the first stage, we implement a multi-task learning structure to segment the surgical limbus region and extract limbus region-focused spatial feature for each frame. In the second stage, we propose the long-short spatiotemporal aggregation transformer (LS-SAT) network to model local fine-grained and global temporal relationships, and combine the extracted spatial features to recognize the current surgical phase. Additionally, we collaborate closely with ophthalmologists to design AR visual cues by utilizing techniques such as limbus ellipse fitting and regional restricted normal cross-correlation rotation computation. We evaluated the network on publicly available and in-house datasets, with comparison results demonstrating its superior performance compared to related works. Ablation results further validated the effectiveness of the limbus region-focused spatial feature extractor and the combination of temporal features. Furthermore, the developed system was evaluated in a clinical setup, with results indicating remarkable accuracy and real-time performance. underscoring its potential for clinical applications.

Computer Vision and Pattern Recognition
Quantification of Robotic Surgeries with Vision-Based Deep Learning

Dani Kiyasseh,Runzhuo Ma,Taseen F. Haque,Jessica Nguyen,Christian Wagner,Animashree Anandkumar,Andrew J. Hung

DOI: https://doi.org/10.48550/arXiv.2205.03028

2022-05-06

Abstract:Surgery is a high-stakes domain where surgeons must navigate critical anatomical structures and actively avoid potential complications while achieving the main task at hand. Such surgical activity has been shown to affect long-term patient outcomes. To better understand this relationship, whose mechanics remain unknown for the majority of surgical procedures, we hypothesize that the core elements of surgery must first be quantified in a reliable, objective, and scalable manner. We believe this is a prerequisite for the provision of surgical feedback and modulation of surgeon performance in pursuit of improved patient outcomes. To holistically quantify surgeries, we propose a unified deep learning framework, entitled Roboformer, which operates exclusively on videos recorded during surgery to independently achieve multiple tasks: surgical phase recognition (the what of surgery), gesture classification and skills assessment (the how of surgery). We validated our framework on four video-based datasets of two commonly-encountered types of steps (dissection and suturing) within minimally-invasive robotic surgeries. We demonstrated that our framework can generalize well to unseen videos, surgeons, medical centres, and surgical procedures. We also found that our framework, which naturally lends itself to explainable findings, identified relevant information when achieving a particular task. These findings are likely to instill surgeons with more confidence in our framework's behaviour, increasing the likelihood of clinical adoption, and thus paving the way for more targeted surgical feedback.

Robotics,Computer Vision and Pattern Recognition,Machine Learning
Computer-Aided Intraoperative Toric Intraocular Lens Positioning and Alignment During Cataract Surgery

Yuxuan Zhai,Guanghua Zhang,Longsheng Zheng,Guangqian Yang,Ke Zhao,Yubin Gong,Zhe Zhang,Ximei Zhang,Bin Sun,Zhao Wang

DOI: https://doi.org/10.1109/jbhi.2021.3072246

IF: 7.7

2021-10-01

IEEE Journal of Biomedical and Health Informatics

Abstract:Cataract causes more than half of all blindness worldwide. The most effective treatment is surgery, where cataract is often replaced by intraocular lens (IOL). Beyond saving vision, toric IOL implantation is becoming increasingly popular to correct corneal astigmatism. It is important to precisely position and align the axis of IOL during surgery to achieve optimal post-operative astigmatism correction. Comparing with conventional manual marking, automated markerless IOL alignment can be faster, more accurate and non-invasive. Here we propose a framework for computer-assisted intraoperative IOL positioning and alignment based on detection and tracking. Firstly, the iris boundary was segmented and the eye center was determined. A statistical sampling method was developed to segment iris and generate training labels, and both conventional algorithms and deep convolutional neural network (CNN) methods were evaluated. Then, regions of interests (ROIs) containing high density of scleral capillaries were used for tracking eye rotations. Both correlation filter and CNN methods were evaluated for tracking. Cumulative errors during long-term tracking were corrected using a reference image. Validation studies against manual labeling using 7 clinical cataract surgical videos demonstrated that the proposed algorithm achieved an average position error around 0.2 mm, an axis alignment error of < 1 <sup>°</sup>, and a frame rate of > 25 FPS, and can be potentially used intraoperatively for markerless IOL positioning and alignment during cataract surgery.

computer science, interdisciplinary applications,mathematical & computational biology,medical informatics, information systems
Towards Deep Learning Guided Autonomous Eye Surgery Using Microscope and iOCT Images

Ji Woong Kim,Shuwen Wei,Peiyao Zhang,Peter Gehlbach,Jin U. Kang,Iulian Iordachita,Marin Kobilarov

2023-07-27

Abstract:Recent advancements in retinal surgery have paved the way for a modern operating room equipped with a surgical robot, a microscope, and intraoperative optical coherence tomography (iOCT)- a depth sensor widely used in retinal surgery. Integrating these tools raises the fundamental question of how to effectively combine them to enable surgical autonomy. In this work, we tackle this question by developing a unified framework that facilitates real-time autonomous surgical workflows leveraging these devices. The system features: (1) a novel imaging system that integrates the microscope and iOCT in real-time by dynamically tracking the surgical instrument via a small iOCT scanning region, providing real-time depth feedback; (2) implementation of convolutional neural networks (CNN) that automatically detect and segment task-relevant information for surgical autonomy; (3) intuitive selection of goal waypoints within both the microscope and iOCT views through simple mouse-click interactions; and (4) integration of model predictive control (MPC) for trajectory generation, ensuring patient safety by implementing safety-related kinematic constraints. The system's utility is demonstrated by automating subretinal injection (SI), a challenging procedure with high accuracy and depth perception requirements. We validate our system by conducting 30 successful SI trials on pig eyes, achieving mean needle insertion accuracy of 26 micrometers to various subretinal goals and mean duration of 55 seconds. Preliminary comparisons to a human operator performing SI in robot-assisted mode highlight the enhanced safety of our system. Project website is here: <a class="link-external link-https" href="https://sites.google.com/view/eyesurgerymicroscopeoct/home" rel="external noopener nofollow">this https URL</a>

Robotics
SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

Yang Li,Florian Richter,Jingpei Lu,Emily K. Funk,Ryan K. Orosco,Jianke Zhu,Michael C. Yip

DOI: https://doi.org/10.1109/LRA.2020.2970659

2020-02-14

Abstract:Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.

Robotics,Computer Vision and Pattern Recognition,Image and Video Processing
SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

Jingpei Lu,Ambareesh Jayakumari,Florian Richter,Yang Li,Michael C. Yip

2021-03-25

Abstract:Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and reduced feature engineering efforts to fully perceive a surgical scene. The framework was tested on three publicly available datasets, which use the da Vinci Surgical System, for comprehensive analysis. Experimental results show that our framework achieves state-of-the-art tracking performance in a surgical environment by utilizing deep learning for feature extraction.

Robotics,Computer Vision and Pattern Recognition
Hyaluronan synthesis by epiphysial chondrocytes is regulated by growth hormone, insulin-like growth factor-1, parathyroid hormone and transforming growth factor-beta 1.

Prasit Pavasant,T. Shizari,Charles B. Underhill

DOI: https://doi.org/10.1016/S0945-053X(96)90161-4

IF: 10.447

1996-12-01

Matrix Biology

Abstract:
Vision-Guided Autonomous Surgical Subtasks via Surgical Robots with Artificial Intelligence

Changyeob Shin

Abstract:Author(s): Shin, Changyeob | Advisor(s): Rosen, Jacob | Abstract: The introduction of automation into surgery may redefine the role of surgeons in operating rooms. While the majority of the manipulation will be performed autonomously by surgical robots, the surgeons may focus on decision-making procedures. This will drastically reduce the burden to surgeons by allowing them to instead interpret the abundant and intelligent information from the system, and will enhance the surgical outcome. To introduce the automation into surgery, the surgical robots are required to have: 1) high precision, 2) motion planning capabilities, and 3) scene understanding. Currently, surgical robots are commonly designed as cable-driven due to safety and several benefits such as low inertia. However, the cable-driven system has low precision because of cable stretch and long chains of cables. Therefore, a new control scheme of cable-driven surgical robots should be developed to overcome these limitations. Surgery is a complicated task consisting of multiple subtasks. To achieve the intermediate steps, motion planner should be developed. In surgery, the manipulation target objects are mostly soft tissue which introduces challenges in modeling the dynamics between the tool and the soft tissue. The motion planner should deal with the unknown dynamics while accomplishing each task. The surgical environment is further complicated by the many blood-covered anatomical structures. Surgeons use the visual feedback through an endoscope camera or other imaging devices, which provide rich information. Although the imaging devices are useful in understanding the surrounding anatomy, images from the devices are high-dimensional and it is difficult to process using algorithms to get high-level information. Therefore, vision-based perception algorithms to understand the relevant anatomy should be developed.This dissertation addresses the three problems above. In chapter two, a hybrid control scheme which utilizes both model-based and data-driven methods is introduced to improve the precision of the cable-driven surgical robots and robustness to hand-eye calibration errors. The convergence of the controller is shown theoretically and experimentally with the Raven IV. Additionally, the efficacy of the controller to clinical tasks is shown by demonstrating the autonomous operations of needle transfer and tissue debridement tasks. In chapter three, learning-based path planning algorithms are proposed for autonomous soft tissue manipulation. The planning algorithms learn the dynamics between the motion of a surgical tool and soft tissue, and the internal controller uses the learned dynamics to manipulate the soft tissue. The performance of developed algorithms is verified on a designed simulation and a robot experiment with the Raven IV. In chapter four, the semantic segmentation algorithm of the optical coherence tomography images for the automated lens extraction is presented. The algorithm uses the deep learning method and provides the capability of understanding the cross-sectional view of the eye anatomy. Furthermore, this segmentation algorithm is incorporated into the Intraocular Robotic Interventional and Surgical System (IRISS) to realize the semi-autonomous lens removal. The experimental results on 7 ex vivo pig eyes verified the efficacy of the developed framework.

Medicine,Engineering,Computer Science
Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

Khusniddin Fozilov,Jacinto Colan,Ana Davila,Kazunari Misawa,Jie Qiu,Yuichiro Hayashi,Kensaku Mori,Yasuhisa Hasegawa

DOI: https://doi.org/10.3390/s23249865

2023-12-16

Abstract:In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system's robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system's potential for real-world application.
Dynamic Detection of Global Microfeatures in Multimodal Retinal Images

Zhen Li,Yawen Deng,Gui-Bin Bian,Weihong Yu,Zhangguo Yu

DOI: https://doi.org/10.1109/tim.2024.3470950

IF: 5.6

2024-10-11

IEEE Transactions on Instrumentation and Measurement

Abstract:Vitreoretinal diseases cause vision disorders and complete vision loss, presenting challenges for surgeons due to operating in dimly lit and complex spaces. Robot-assisted surgery enhances precision (PC) and efficiency through high repeatability and advanced imaging. However, preoperative autonomous navigation methods face blurred boundaries, element confusion, and occlusions when detecting intraoperative retinal microfeatures. Their core deficiency lies in handling large-scale variations, high local similarity, and insufficient feature extraction during the transition to intraoperative. Therefore, a detection framework for full-vision intraoperative retinal microfeatures is established, utilizing multimodal image fusion, dynamic tracking, and feature sharing, consisting of feature segmentation and matching. A novel Transformer-based U-Net (TransUNet) model with multidepth partial convolutional residual layers and edge dilation is proposed for intraoperative fundus image segmentation, refining and sharpening microfeatures for clear boundary delineation. Subsequently, a multiscale, two-stage microfeature matching method fusion of preoperative and intraoperative images accurately completes features in instrument-occluded and invisible areas. A dynamic keypoints' tracking and updating mechanism for sequential images overcomes element confusion by inheritance and refreshing contextual information. Experimental results show that the proposed framework can detect retinal microfeatures with an accuracy (ACC) of 90.47%, which is 15.76% higher than intraoperative segmentation. Ideally, it can complement more than 99.70% of features in the instruments' occluded region and more than 97.47% in the invisible region when the visible features' area reaches more than 1200 pixels. The proposed framework provides reliable and stable intraoperative reference points for robot localization, significantly enhancing perception and understanding of the environment in robot-assisted vitreoretinal surgeries.

engineering, electrical & electronic,instruments & instrumentation
A Computational Framework for Intraoperative Pupil Analysis in Cataract Surgery

Binh Duong Giap,Karthik Srinivasan,Ossama Mahmoud,Dena Ballouz,Jefferson Lustre,Keely Likosky,Shahzad I Mian,Bradford L Tannen,Nambi Nallasamy

DOI: https://doi.org/10.1016/j.xops.2024.100597

2024-08-22

Abstract:Purpose: Pupillary instability is a known risk factor for complications in cataract surgery. This study aims to develop and validate an innovative and reliable computational framework for the automated assessment of pupil morphologic changes during the various phases of cataract surgery. Design: Retrospective surgical video analysis. Subjects: Two hundred forty complete surgical video recordings, among which 190 surgeries were conducted without the use of pupil expansion devices (PEDs) and 50 were performed with the use of a PED. Methods: The proposed framework consists of 3 stages: feature extraction, deep learning (DL)-based anatomy recognition, and obstruction (OB) detection/compensation. In the first stage, surgical video frames undergo noise reduction using a tensor-based wavelet feature extraction method. In the second stage, DL-based segmentation models are trained and employed to segment the pupil, limbus, and palpebral fissure. In the third stage, obstructed visualization of the pupil is detected and compensated for using a DL-based algorithm. A dataset of 5700 intraoperative video frames across 190 cataract surgeries in the BigCat database was collected for validating algorithm performance. Main outcome measures: The pupil analysis framework was assessed on the basis of segmentation performance for both obstructed and unobstructed pupils. Classification performance of models utilizing the segmented pupil time series to predict surgeon use of a PED was also assessed. Results: An architecture based on the Feature Pyramid Network model with Visual Geometry Group 16 backbone integrated with the adaptive wavelet tensor feature extraction feature extraction method demonstrated the highest performance in anatomy segmentation, with Dice coefficient of 96.52%. Incorporation of an OB compensation algorithm improved performance further (Dice 96.82%). Downstream analysis of framework output enabled the development of a Support Vector Machine-based classifier that could predict surgeon usage of a PED prior to its placement with 96.67% accuracy and area under the curve of 99.44%. Conclusions: The experimental results demonstrate that the proposed framework (1) provides high accuracy in pupil analysis compared with human-annotated ground truth, (2) substantially outperforms isolated use of a DL segmentation model, and (3) can enable downstream analytics with clinically valuable predictive capacity. Financial disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

Jianqing Peng,Chi Zhang,Liang Kang,Guodong Feng

DOI: https://doi.org/10.1109/tim.2022.3204086

IF: 5.6

2022-10-01

IEEE Transactions on Instrumentation and Measurement

Abstract:In laparoscopic surgeries, intelligent-assisted endoscope-holding robots (AEHRs) with traditional servo control have limited performance due to the lack of endoscopic attitude control, hand–eye coordination, and image definition. This article proposes an endoscope field-of-view (FOV) autonomous tracking method for robot-assisted surgery (RAS) considering pose (i.e., position and attitude) control, hand–eye coordination, and image definition, and it can realize that the AEHR system can safely, stably, and quickly track the movement of surgical instruments. First, the general framework of the intelligent tracking system with robot-assisted endoscope holding is constructed according to the clinical needs and shortcomings of traditional research. Second, the kinematic model and remote center of motion (RCM) constraint equations of AEHRs are established. Then, surgical instrument tips are segmented and localized using the deep learning method (DLM), and further, a surgical FOV intelligent adjustment control model with multiple indexes is developed. Finally, an experimental system based on the hybrid of AEHRs and speech interaction is built. The simulation and experimental results demonstrate that the proposed method can achieve fast, accurate, and effective instrument tips tracking (ITT), reduce hand–eye incoordination, and improve the safety and stability of FOV intelligent adjustment.

engineering, electrical & electronic,instruments & instrumentation
Vitreoretinal Surgical Instrument Tracking in Three Dimensions Using Deep Learning

Pierre F. Baldi,Sherif Abdelkarim,Junze Liu,Josiah K. To,Marialejandra Diaz Ibarra,Andrew W. Browne

DOI: https://doi.org/10.1167/tvst.12.1.20

2023-01-17

Translational Vision Science & Technology

Abstract:Purpose: To evaluate the potential for artificial intelligence-based video analysis to determine surgical instrument characteristics when moving in the three-dimensional vitreous space.Methods: We designed and manufactured a model eye in which we recorded choreographed videos of many surgical instruments moving throughout the eye. We labeled each frame of the videos to describe the surgical tool characteristics: tool type, location, depth, and insertional laterality. We trained two different deep learning models to predict each of the tool characteristics and evaluated model performances on a subset of images.Results: The accuracy of the classification model on the training set is 84% for the x-y region, 97% for depth, 100% for instrument type, and 100% for laterality of insertion. The accuracy of the classification model on the validation dataset is 83% for the x-y region, 96% for depth, 100% for instrument type, and 100% for laterality of insertion. The close-up detection model performs at 67 frames per second, with precision for most instruments higher than 75%, achieving a mean average precision of 79.3%.Conclusions: We demonstrated that trained models can track surgical instrument movement in three-dimensional space and determine instrument depth, tip location, instrument insertional laterality, and instrument type. Model performance is nearly instantaneous and justifies further investigation into application to real-world surgical videos.Translational Relevance: Deep learning offers the potential for software-based safety feedback mechanisms during surgery or the ability to extract metrics of surgical technique that can direct research to optimize surgical outcomes.

ophthalmology
EyeLS: Shadow-Guided Instrument Landing System for Intraocular Target Approaching in Robotic Eye Surgery

Junjie Yang,Zhihao Zhao,Siyuan Shen,Daniel Zapp,Mathias Maier,Kai Huang,Nassir Navab,M. Ali Nasseri

DOI: https://doi.org/10.48550/arXiv.2311.08799

2023-11-15

Abstract:Robotic ophthalmic surgery is an emerging technology to facilitate high-precision interventions such as retina penetration in subretinal injection and removal of floating tissues in retinal detachment depending on the input imaging modalities such as microscopy and intraoperative OCT (iOCT). Although iOCT is explored to locate the needle tip within its range-limited ROI, it is still difficult to coordinate iOCT's motion with the needle, especially at the initial target-approaching stage. Meanwhile, due to 2D perspective projection and thus the loss of depth information, current image-based methods cannot effectively estimate the needle tip's trajectory towards both retinal and floating targets. To address this limitation, we propose to use the shadow positions of the target and the instrument tip to estimate their relative depth position and accordingly optimize the instrument tip's insertion trajectory until the tip approaches targets within iOCT's scanning area. Our method succeeds target approaching on a retina model, and achieves an average depth error of 0.0127 mm and 0.3473 mm for floating and retinal targets respectively in the surgical simulator without damaging the retina.

Robotics,Computer Vision and Pattern Recognition

Capsaicin-induced biphasic oxygen uptake in rat muscle: antagonism by capsazepine and ruthenium red provides further evidence for peripheral vanilloid receptor subtypes (VN1/VN2).

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation

DeepPhase: Surgical Phase Recognition in CATARACTS Videos

Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Swept Source Intraoperative OCT Angiography

Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Computer-Aided Intraoperative Toric Intraocular Lens Positioning and Alignment During Cataract Surgery

Towards Deep Learning Guided Autonomous Eye Surgery Using Microscope and iOCT Images

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

Hyaluronan synthesis by epiphysial chondrocytes is regulated by growth hormone, insulin-like growth factor-1, parathyroid hormone and transforming growth factor-beta 1.

Vision-Guided Autonomous Surgical Subtasks via Surgical Robots with Artificial Intelligence

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

Dynamic Detection of Global Microfeatures in Multimodal Retinal Images

A Computational Framework for Intraoperative Pupil Analysis in Cataract Surgery

Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

Vitreoretinal Surgical Instrument Tracking in Three Dimensions Using Deep Learning

EyeLS: Shadow-Guided Instrument Landing System for Intraocular Target Approaching in Robotic Eye Surgery