Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields

A Three-Dimensional Measurement Method for Binocular Endoscopes Based on Deep Learning

Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling

EndoPerfect: A Hybrid NeRF-Stereo Vision Approach Pioneering Monocular Depth Estimation and 3D Reconstruction in Endoscopy

Self-supervised neural network-based endoscopic monocular 3D reconstruction method

Neural radiance fields-based multi-view endoscopic scene reconstruction for surgical simulation

Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery

BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields

Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Enhancing endoscopic scene reconstruction with color-aware inverse rendering through neural SDF and radiance fields

A Review of 3D Reconstruction Techniques for Deformable Tissues in Robotic Surgery

Monocular endoscopy images depth estimation with multi-scale residual fusion

Efficient EndoNeRF Reconstruction and Its Application for Data-driven Surgical Simulation

Multi-scale, Multi-Dimensional Binocular Endoscopic Image Depth Estimation Network.

Dynamic Depth-Supervised NeRF for Multi-View RGB-D Operating Room Images

Unsupervised-Learning-Based Continuous Depth and Motion Estimation with Monocular Endoscopy for Virtual Reality Minimally Invasive Surgery

ENeRF-SLAM:A Dense Endoscopic SLAM with Neural Implicit Representation