Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Intra-operative Monocular 3D Reconstruction for Image-Guided Navigation in Active Locomotion Capsule Endoscopy

Incorporation of 3D Model and Panoramic View for Gastroscopic Lesion Surveillance.

Autonomous Magnetic Navigation Framework for Active Wireless Capsule Endoscopy Inspired by Conventional Colonoscopy Procedures

Sparse-then-Dense Alignment based 3D Map Reconstruction Method for Endoscopic Capsule Robots

Hybrid Magnetic and Vision Localization Technique of Capsule Endoscope for 3d Recovery of Pathological Tissues

Single-Image-Based 3D Reconstruction of Endoscopic Images

A Time-of-Flight Stereoscopic Endoscope for Anatomical 3D Reconstruction

Autonomous Intelligent Navigation for Flexible Endoscopy Using Monocular Depth Guidance and 3-D Shape Planning

Photometric single-view dense 3D reconstruction in endoscopy

Capsule Endoscopy: From Current Achievements to Open Challenges

A fully dense and globally consistent 3D map reconstruction approach for GI tract to enhance therapeutic relevance of the endoscopic capsule robot

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots

Surface Reconstruction from Tracked Endoscopic Video Using the Structure from Motion Approach.

Endoscope As A Tracking Tool For Computer Assisted Minimally Invasive Ventral Spine Surgeries

Video-based measurements for wireless capsule endoscope tracking

Magnetic-Visual Sensor Fusion-based Dense 3D Reconstruction and Localization for Endoscopic Capsule Robots

Robot-assisted magnetic capsule endoscopy; navigating colorectal inclinations

BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications

Endoscope-based Hybrid Navigation System for Minimally Invasive Ventral Spine Surgeries.

Improved Multiple Objects Tracking Based Autonomous Simultaneous Magnetic Actuation & Localization for WCE