Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Ultrasound Image-Guided Mapping of Endoscopic Views on a 3D Placenta Model: A Tracker-Less Approach.

Ultrasound Image-Based Endoscope Localization for Minimally Invasive Fetoscopic Surgery.

Fast Image Mapping of Endoscopic Image Mosaics with Three-Dimensional Ultrasound Image for Intrauterine Fetal Surgery.

Incorporation of 3D Model and Panoramic View for Gastroscopic Lesion Surveillance.

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery

A Simplified 3D Ultrasound Freehand Imaging Framework Using 1D Linear Probe and Low-Cost Mechanical Track

Real-time Surgical Tool Tracking and Pose Estimation Using a Hybrid Cylindrical Marker.

Freehand 3D Ultrasound Imaging Based on Probe-mounted Vision and IMU System

EchoFusion: Tracking and Reconstruction of Objects in 4D Freehand Ultrasound Imaging without External Trackers

Gpu-Based Fast 3d Ultrasound-Endoscope Image Fusion For Complex-Shaped Objects

Adaptive 3D Localization of 2D Freehand Ultrasound Brain Images

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

From 2D Ultrasound to Patient-Specific 3D Surface Models for Interventional Guidance

Robot-assisted Ultrasound-guided Tracking of Anatomical Structures for the Application of Focused Ultrasound

Online 4D Ultrasound-Guided Robotic Tracking Enables 3D Ultrasound Localisation Microscopy with Large Tissue Displacements

Spatial Position Measurement System for Surgical Navigation Using 3-D Image Marker-Based Tracking Tools with Compact Volume

Automatic and Robust Freehand Ultrasound Calibration Using a Tracked Pointer

Markerless Inside-Out Tracking for Interventional Applications

An EM-Tracked Approach for Calibrating the 3D Pose of Flexible Endoscopes

A Time-of-Flight Stereoscopic Endoscope for Anatomical 3D Reconstruction

Photometric single-view dense 3D reconstruction in endoscopy