Online 3D reconstruction and dense tracking in endoscopic videos

Michel Hayoz,Christopher Hahne,Thomas Kurmann,Max Allan,Guido Beldi,Daniel Candinas,ablo Márquez-Neila,Raphael Sznitman

2024-09-10

Abstract:3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. In this work, we present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions. Our method dynamically extends a canonical scene representation using Gaussian splatting, while modeling tissue deformations through a sparse set of control points. We introduce an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction. Through experiments on the StereoMIS dataset, we demonstrate the effectiveness of our approach, outperforming state-of-the-art tracking methods and achieving comparable performance to offline reconstruction techniques. Our work enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve online 3D scene reconstruction and dense tracking through stereo - endoscope video data, so as to enhance the understanding of the surgical scene and assist surgical interventions. Specifically, the author proposes a new framework, aiming to dynamically expand the canonical scene representation and model tissue deformation through a set of sparse control points, thereby achieving consistent tracking and accurate reconstruction.

### Main Problems

1. **Real - time and Consistency**: Current methods either rely on offline processing or lack physical constraints to control tissue deformation, and are unable to achieve real - time and consistent 3D estimation.
2. **Handling of Complex Surgical Scenes**: There are challenges such as respiratory movement, tissue deformation and occlusion in the surgical scene, and a method that can handle these complex situations is required.
3. **Requirement for Dense Tracking**: Existing methods mostly focus on sparse point tracking, while downstream applications (such as augmented reality overlay and robot - assisted) usually require dense tracking.

### Solutions

The author proposes an online framework based on Gaussian splatting, and the main contributions include:

- **Dynamic Scene Expansion**: As new parts of the scene appear, new Gaussian models are gradually added to ensure the integrity and accuracy of the scene representation.
- **Tissue Deformation Modeling**: Model tissue deformation through a set of sparse control points, use Gaussian kernel interpolation to reduce the number of control points in the deformation field, improve the fitting speed and simplify the geometric prior.
- **Efficient Online Fitting Algorithm**: Optimize the scene parameters to make the tracking and reconstruction consistent and accurate.
- **Optical Flow Initialization**: Use optical flow to initialize the motion in a single - camera setup to accelerate convergence.

### Experimental Verification

The author verifies the effectiveness of this method through experiments on the StereoMIS dataset, demonstrating its superior performance in point tracking, 3D reconstruction, etc., outperforming the existing state - of - the - art tracking methods and being comparable to the performance of offline reconstruction techniques.

### Downstream Applications

This method also shows its potential in downstream tasks such as 3D semantic segmentation, highlighting its application prospects in surgical training, augmented reality overlay and robot - assisted, etc.

### Summary

This paper achieves online 3D scene reconstruction and dense tracking from stereo - endoscope video data by introducing Gaussian splatting and sparse control point modeling, solves the deficiencies of existing methods in terms of real - time and complex scene handling, and provides new ideas and technical support for the development of surgical - assistance systems.