Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

A lumen-adapted navigation scheme with spatial awareness from monocular vision for autonomous robotic endoscopy

Autonomous Intraluminal Navigation of a Soft Robot using Deep-Learning-based Visual Servoing

Autonomous Magnetic Navigation Framework for Active Wireless Capsule Endoscopy Inspired by Conventional Colonoscopy Procedures

Multistage adaptive control strategy based on image contour data for autonomous endoscope navigation

Autonomous Magnetic Navigation in Endoscopic Image Mosaics

Semantic SLAM Based on Deep Learning in Endocavity Environment

Incorporation of 3D Model and Panoramic View for Gastroscopic Lesion Surveillance.

Toward Human-Out-of-the-Loop Endoscope Navigation Based on Context Awareness for Enhanced Autonomy in Robotic Surgery

Spatially Compact Visual Navigation System for Automated Suturing Robot Towards Oral and Maxillofacial Surgery

Slice-based Virtual Endoscopy Navigation

Spatially Compact Visual Navigation System for Automated Suturing Robot Toward Oral and Maxillofacial Surgery

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

The Future of Endoscopic Navigation: A Review of Advanced Endoscopic Vision Technology

Augmented‐reality‐based surgical navigation for endoscope retrograde cholangiopancreatography: A phantom study

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery

A navigation system for minimally invasive abdominal intervention surgery robot

Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study

Safety-aware robotic steering of a flexible endoscope for nasotracheal intubation

Motor-free telerobotic endomicroscopy for steerable and programmable imaging in complex curved and localized areas

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection

A geometry-aware deep network for depth estimation in monocular endoscopy