Abstract:State-of-the-art research of traditional computer vision is increasingly leveraged in the surgical domain. A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation using deep-learning methods. However, state-of-the-art single-view pose estimation methods do not yet meet the accuracy required for surgical navigation. In this context, we investigate the benefits of multi-view setups for highly accurate and occlusion-robust 6DoF pose estimation of surgical instruments and derive recommendations for an ideal camera system that addresses the challenges in the operating room. The contributions of this work are threefold. First, we present a multi-camera capture setup consisting of static and head-mounted cameras, which allows us to study the performance of pose estimation methods under various camera configurations. Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre and including rich annotations for surgeon, instrument, and patient anatomy. Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments and analyze the influence of camera configurations, training data, and occlusions on the pose accuracy and generalization ability. The best method utilizes five cameras in a multi-view pose optimization and achieves an average position and orientation error of 1.01 mm and 0.89°for a surgical drill as well as 2.79 mm and 3.33°for a screwdriver under optimal conditions. Our results demonstrate that marker-less tracking of surgical instruments is becoming a feasible alternative to existing marker-based systems.

Single-Shot Pose Estimation of Surgical Robot Instruments' Shafts from Monocular Endoscopic Images

MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization

Monocular pose estimation of articulated surgical instruments in open surgery

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments

Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery

Evaluation of single-stage vision models for pose estimation of surgical instruments

Weakly Supervised Pose Estimation of Surgical Instrument from a Single Endoscopic Image

Pose Quality Prediction for Vision Guided Robotic Shoulder Arthroplasty.

Advancing 6-DoF Instrument Pose Estimation in Variable X-Ray Imaging Geometries

Using hand pose estimation to automate open surgery training feedback

Realistic Data Generation for 6D Pose Estimation of Surgical Instruments

Pose Estimation Via Structure-Depth Information from Monocular Endoscopy Images Sequence.

Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network

Learning How To Robustly Estimate Camera Pose in Endoscopic Videos

Towards Robotic Eye Surgery: Marker-free, Online Hand-eye Calibration using Optical Coherence Tomography Images

SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Quantification of Robotic Surgeries with Vision-Based Deep Learning