Abstract:Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geometric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demonstrate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

A Three-Dimensional Measurement Method for Binocular Endoscopes Based on Deep Learning

Three-Dimensional Stitching of Binocular Endoscopic Images Based on Feature Points

Research progress of depth detection in vision measurement: A novel project of bifocal imaging system for 3D measurement

Binocular Depth Estimation Using Convolutional Neural Network With Siamese Branches.

Using A Bi-Prism Endoscopic System for Three-Dimensional Measurement

Robust Cost Volume Generation Method for Dense Stereo Matching in Endoscopic Scenarios

Three-Dimensional Point Cloud Reconstruction Method of Cardiac Soft Tissue Based on Binocular Endoscopic Images

3D endoscopic depth estimation using 3D surface-aware constraints

3D reconstruction from endoscopy images: A survey

Multi-scale, Multi-Dimensional Binocular Endoscopic Image Depth Estimation Network.

Real-Time Dense Reconstruction with Binocular Endoscopy Based on StereoNet and ORB-SLAM

Self-Supervised Micro-Baseline Binocular Endoscope Depth Estimation Method with Domain Adaptation

Taking measurement in every direction: Implicit scene representation for accurately estimating target dimensions under monocular endoscope

EndoPerfect: A Hybrid NeRF-Stereo Vision Approach Pioneering Monocular Depth Estimation and 3D Reconstruction in Endoscopy

Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

Binocular Visual Measurement Method Based on Feature Matching

An Endoscope Designed with 3d Measurement Functions

Three-Dimensional Measurement Approach in Small FOV and Confined Space Using an Electronic Endoscope

Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling

Deep Convolutional Network for Stereo Depth Mapping in Binocular Endoscopy

Self-supervised neural network-based endoscopic monocular 3D reconstruction method