Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery

Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery

EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera

DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model

Learning to Adapt Foundation Model DINOv2 for Capsule Endoscopy Diagnosis

Surgical Depth Anything: Depth Estimation for Surgical Scenes using Foundation Models

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

3D endoscopic depth estimation using 3D surface-aware constraints

Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy

Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery

EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels

MonoLoT: Self-Supervised Monocular Depth Estimation in Low-Texture Scenes for Automatic Robotic Endoscopy

Self-Supervised Depth Estimation in Laparoscopic Image using 3D Geometric Consistency

ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation

Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images

Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images

Towards Full-parameter and Parameter-efficient Self-learning For Endoscopic Camera Depth Estimation

Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling

Self-supervised Monocular Depth Estimation with 3D Displacement Module for Laparoscopic Images