Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Context-Aware Depth and Pose Estimation for Bronchoscopic Navigation.

Robust camera localisation with depth reconstruction for bronchoscopic navigation

DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy

A Robust Pose Optimization Scheme With Spatial Geometry Awareness for Hybrid Bronchoscopic Navigation

A bronchoscopic navigation method based on neural radiation fields

Generative Localization with Uncertainty Estimation Through Video-CT Data for Bronchoscopic Biopsy.

Landmark Based Bronchoscope Localization for Needle Insertion Under Respiratory Deformation

PANS: Probabilistic Airway Navigation System for Real-time Robust Bronchoscope Localization

Enhancing Bronchoscopy Depth Estimation through Synthetic-to-Real Domain Adaptation

Self-supervised Monocular Depth and Pose Estimation for Endoscopy with Generative Latent Priors

Airway label prediction in video bronchoscopy: capturing temporal dependencies utilizing anatomical knowledge

BRANCH: Bifurcation Recognition for Airway Navigation based on struCtural cHaracteristics.

SLAM Endoscopy enhanced by adversarial depth prediction

Weakly Supervised Airway Orifice Segmentation in Video Bronchoscopy

Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy

Task-guided Domain Gap Reduction for Monocular Depth Prediction in Endoscopy

Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy

MonoLoT: Self-Supervised Monocular Depth Estimation in Low-Texture Scenes for Automatic Robotic Endoscopy

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery