Abstract:Background and objective: Surgical robotics tends to develop cognitive control architectures to provide certain degree of autonomy to improve patient safety and surgery outcomes, while decreasing the required surgeons' cognitive load dedicated to low level decisions. Cognition needs workspace perception, which is an essential step towards automatic decision-making and task planning capabilities. Robust and accurate detection and tracking in minimally invasive surgery suffers from limited visibility, occlusions, anatomy deformations and camera movements. Method: This paper develops a robust methodology to detect and track anatomical structures in real time to be used in automatic control of robotic systems and augmented reality. The work focuses on the experimental validation in highly challenging surgery: fetoscopic repair of Open Spina Bifida. The proposed method is based on two sequential steps: first, selection of relevant points (contour) using a Convolutional Neural Network and, second, reconstruction of the anatomical shape by means of deformable geometric primitives. Results: The methodology performance was validated with different scenarios. Synthetic scenario tests, designed for extreme validation conditions, demonstrate the safety margin offered by the methodology with respect to the nominal conditions during surgery. Real scenario experiments have demonstrated the validity of the method in terms of accuracy, robustness and computational efficiency. Conclusions: This paper presents a robust anatomical structure detection in present of abrupt camera movements, severe occlusions and deformations. Even though the paper focuses on a case study, Open Spina Bifida, the methodology is applicable in all anatomies which contours can be approximated by geometric primitives. The methodology is designed to provide effective inputs to cognitive robotic control and augmented reality systems that require accurate tracking of sensitive anatomies.

Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives

SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation

Real-time 3D Tracking of Articulated Tools for Robotic Surgery

Adaptive Image-Based Positioning of RCM Mechanisms Using Angle and Distance Features

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Gaze-Contingent Soft Tissue Deformation Tracking for Minimally Invasive Robotic Surgery

A dual-instrument Kalman-based tracker to enhance robustness of microsurgical tools tracking

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization

Probabilistic tracking of affine-invariant anisotropic regions.

Robust tracking of deformable anatomical structures with severe occlusions using deformable geometrical primitives

A Markerless 3D Tracking Framework for Continuum Surgical Tools Using a Surgical Tool Partial Pose Estimation Network Based on Domain Randomization

ST-ITEF: Spatio-Temporal Intraoperative Task Estimating Framework to recognize surgical phase and predict instrument path based on multi-object tracking in keratoplasty

Real-time Surgical Tool Tracking and Pose Estimation Using a Hybrid Cylindrical Marker.

Dynamic guidance for robotic surgery using image-constrained biomechanical models.

Robust hand tracking for surgical telestration

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model.

Real-Time Medical Tool Runout Monitor Based on Dual Laser Displacement Sensors

Robust Image-Based Computation of the 3D Position of RCM Instruments and Its Application to Image-Guided Manipulation

Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments

Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries