Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting

Shuojue Yang,Qian Li,Daiyun Shen,Bingchen Gong,Qi Dou,Yueming Jin

2024-05-30

Abstract:Tissue deformation poses a key challenge for accurate surgical scene reconstruction. Despite yielding high reconstruction quality, existing methods suffer from slow rendering speeds and long training times, limiting their intraoperative applicability. Motivated by recent progress in 3D Gaussian Splatting, an emerging technology in real-time 3D rendering, this work presents a novel fast reconstruction framework, termed Deform3DGS, for deformable tissues during endoscopic surgery. Specifically, we introduce 3D GS into surgical scenes by integrating a point cloud initialization to improve reconstruction. Furthermore, we propose a novel flexible deformation modeling scheme (FDM) to learn tissue deformation dynamics at the level of individual Gaussians. Our FDM can model the surface deformation with efficient representations, allowing for real-time rendering performance. More importantly, FDM significantly accelerates surgical scene reconstruction, demonstrating considerable clinical values, particularly in intraoperative settings where time efficiency is crucial. Experiments on DaVinci robotic surgery videos indicate the efficacy of our approach, showcasing superior reconstruction fidelity PSNR: (37.90) and rendering speed (338.8 FPS) while substantially reducing training time to only 1 minute/scene. Our code is available at <a class="link-external link-https" href="https://github.com/jinlab-imvr/Deform3DGS" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to address the key challenge of tissue deformation in surgical scene reconstruction. Although existing methods can achieve high-quality reconstruction, they often suffer from slow rendering speeds and long training times, which limit their practical application during surgery. To this end, the paper proposes a new framework called Deform3DGS, which accelerates the reconstruction of deformable tissues in surgical scenes by introducing 3D Gaussian Splatting technology.

Specifically, the main contributions of the paper include:
1. **3D Gaussian Point Cloud Initialization**: By combining point cloud initialization techniques, the quality and efficiency of the reconstruction are improved.
2. **Flexible Deformation Modeling Scheme (FDM)**: A new flexible deformation modeling scheme is proposed, which can efficiently represent tissue deformation and support real-time rendering performance.
3. **Efficient Training and Rendering**: Experimental results show that this method significantly reduces training time and improves rendering speed while maintaining high-quality reconstruction, making it particularly suitable for time-efficient intraoperative scenarios.

The paper validates the effectiveness of the method through experiments on the Da Vinci robotic surgery video dataset, demonstrating superior performance in terms of reconstruction quality (PSNR: 37.90) and rendering speed (338.8 FPS), and reducing training time to only 1 minute per scene. These improvements make the method highly valuable for clinical applications.