Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Reconstruct Dynamic Soft-Tissue With Stereo Endoscope Based on a Single-Layer Network

Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery

Fast and Lightweight Network Improves Serial Brain Section Stitching

Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery

Reconstructing a 3D Heart Surface with Stereo-Endoscope by Learning Eigen-Shapes.

Stereo Video Reconstruction Without Explicit Depth Maps for Endoscopic Surgery

Self-Supervised Cyclic Diffeomorphic Mapping for Soft Tissue Deformation Recovery in Robotic Surgery Scenes

Real-time Dense Reconstruction of Tissue Surface from Stereo Optical Video

Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery

Three-Dimensional Point Cloud Reconstruction Method of Cardiac Soft Tissue Based on Binocular Endoscopic Images

Disparity Estimation of Stereo-Endoscopic Images Using Deep Generative Network

EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos

E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception

Real-Time Dense Reconstruction with Binocular Endoscopy Based on StereoNet and ORB-SLAM

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model.

Real-time Surface Deformation Recovery from Stereo Videos

Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Disparity-constrained Stereo Endoscopic Image Super-Resolution.

The algorithm of stereo vision and shape from shading based on endoscope imaging

Self-supervised neural network-based endoscopic monocular 3D reconstruction method