Abstract:Background: Despite the popular use of virtual and physical reality simulators in laparoscopic training, the educational potential of augmented reality (AR) has not received much attention. A major challenge is the robust tracking and three-dimensional (3D) pose estimation of the endoscopic instrument, which are essential for achieving interaction with the virtual world and for realistic rendering when the virtual scene is occluded by the instrument. In this paper we propose a method that addresses these issues, based solely on visual information obtained from the endoscopic camera. Methods: Two different tracking algorithms are combined for estimating the 3D pose of the surgical instrument with respect to the camera. The first tracker creates an adaptive model of a colour strip attached to the distal part of the tool (close to the tip). The second algorithm tracks the endoscopic shaft, using a combined Hough-Kalman approach. The 3D pose is estimated with perspective geometry, using appropriate measurements extracted by the two trackers. Results: The method has been validated on several complex image sequences for its tracking efficiency, pose estimation accuracy and applicability in AR-based training. Using a standard endoscopic camera, the absolute average error of the tip position was 2.5 mm for working distances commonly found in laparoscopic training. The average error of the instrument's angle with respect to the camera plane was approximately 2°. The results are also supplemented by video segments of laparoscopic training tasks performed in a physical and an AR environment. Conclusions: The experiments yielded promising results regarding the potential of applying AR technologies for laparoscopic skills training, based on a computer vision framework. The issue of occlusion handling was adequately addressed. The estimated trajectory of the instruments may also be used for surgical gesture interpretation and assessment.

Extra-abdominal trocar and instrument detection for enhanced surgical workflow understanding

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery

An augmented reality and high-speed optical tracking system for laparoscopic surgery

Reliability of sensor-based real-time workflow recognition in laparoscopic cholecystectomy

Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis

A dual-instrument Kalman-based tracker to enhance robustness of microsurgical tools tracking

Intra-operative surgical instrument usage detection on a multi-sensor table

An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training

Real-time image-based instrument classification for laparoscopic surgery

A Deep Learning Approach for Surgical Instruments Detection System in Total Knee Arthroplasty - Automatic Creation of Training Data and Reduction of Training Time -

Redefining the Laparoscopic Spatial Sense: AI-based Intra- and Postoperative Measurement from Stereoimages

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Real-Time Tool Localization for Laparoscopic Surgery Using Convolutional Neural Network

SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments

Neural Fields for 3D Tracking of Anatomy and Surgical Instruments in Monocular Laparoscopic Video Clips

Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models

Tracking Everything in Robotic-Assisted Surgery

Heidelberg colorectal data set for surgical data science in the sensor operating room

Video-based Surgical Skills Assessment using Long term Tool Tracking

An intelligent and autonomous endoscopic guidance system for minimally invasive surgery