Abstract:Purpose: Recent advances in computer vision and machine learning have resulted in endoscopic video-based solutions for dense reconstruction of the anatomy. To effectively use these systems in surgical navigation, a reliable image-based technique is required to constantly track the endoscopic camera's position within the anatomy, despite frequent removal and re-insertion. In this work, we investigate the use of recent learning-based keypoint descriptors for six degree-of-freedom camera pose estimation in intraoperative endoscopic sequences and under changes in anatomy due to surgical resection. Methods: Our method employs a dense structure from motion (SfM) reconstruction of the preoperative anatomy, obtained with a state-of-the-art patient-specific learning-based descriptor. During the reconstruction step, each estimated 3D point is associated with a descriptor. This information is employed in the intraoperative sequences to establish 2D-3D correspondences for Perspective-n-Point (PnP) camera pose estimation. We evaluate this method in six intraoperative sequences that include anatomical modifications obtained from two cadaveric subjects. Results: Show that this approach led to translation and rotation errors of 3.9 mm and 0.2 radians, respectively, with 21.86% of localized cameras averaged over the six sequences. In comparison to an additional learning-based descriptor (HardNet++), the selected descriptor can achieve a better percentage of localized cameras with similar pose estimation performance. We further discussed potential error causes and limitations of the proposed approach. Conclusion: Patient-specific learning-based descriptors can relocalize images that are well distributed across the inspected anatomy, even where the anatomy is modified. However, camera relocalization in endoscopic sequences remains a persistently challenging problem, and future research is necessary to increase the robustness and accuracy of this technique.

Constrained Visual-Inertial Localization With Application And Benchmark in Laparoscopic Surgery

MITI: SLAM Benchmark for Laparoscopic Surgery

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Investigating keypoint descriptors for camera relocalization in endoscopy surgery

Real-Time Tool Localization for Laparoscopic Surgery Using Convolutional Neural Network

Vision-Based Neurosurgical Guidance: Unsupervised Localization and Camera-Pose Prediction

Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery

Redefining the Laparoscopic Spatial Sense: AI-based Intra- and Postoperative Measurement from Stereoimages

Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments

Geometric Structure Aided Visual Inertial Localization

Study of Visual SLAM Methods in Minimally Invasive Surgery

Learning How To Robustly Estimate Camera Pose in Endoscopic Videos

Markerless Inside-Out Tracking for Interventional Applications

Motion compensated SLAM for image guided surgery.

Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception

Laparoscope self-calibration for robotic assisted minimally invasive surgery.

Tracking monocular camera pose and deformation for SLAM inside the human body

Direct Sparse Monocular Visual-Inertial Odometry with Covisibility Constraints

Vision-based hand-eye calibration for robot-assisted minimally invasive surgery

Visual-based Autonomous Field of View Control of Laparoscope with Safety-Rcm Constraints for Semi-Autonomous Surgery.

Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation