Abstract:Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.

Stereo matching of binocular laparoscopic images with improved densely connected neural architecture search

Human-Computer Interaction System Based on Binocular Stereo Vision

A Three-Dimensional Measurement Method for Binocular Endoscopes Based on Deep Learning

Efficient Binocular Stereo Matching Based on Sad and Improved Census Transformation

Binocular Depth Estimation Using Convolutional Neural Network With Siamese Branches.

Robust Cost Volume Generation Method for Dense Stereo Matching in Endoscopic Scenarios

An accurate stereo matching method based on color segments and edges

Real-Time Dense Reconstruction with Binocular Endoscopy Based on StereoNet and ORB-SLAM

A Disparity Refinement Framework for Learning-based Stereo Matching Methods in Cross-domain Setting for Laparoscopic Images

Exploiting Semantic and Boundary Information for Stereo Matching

Dual-correlate optimized coarse-fine strategy for monocular laparoscopic videos feature matching via multilevel sequential coupling feature descriptor

Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

Stereo matching algorithm using deep learning and edge-preserving filter for machine vision

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Efficient Large Scale Stereo Matching Based on Cross-Scale.

Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation

A Binocular Reconstruction Based on Perspective Projection Constraints and Its Application on Robot Eye‐hand Coordination

Stereo Matching Using Multi-Level Cost Volume and Multi-Scale Feature Constancy

An end-to-end stereo matching algorithm based on improved convolutional neural network

Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images

A Stereo Matching Algorithm for High‐Precision Guidance in a Weakly Textured Industrial Robot Environment Dominated by Planar Facets