Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes

Jiwei Shan,Zeyu Cai,Cheng-Tai Hsieh,Shing Shin Cheng,Hesheng Wang

2025-01-02

Abstract:Efficient and high-fidelity reconstruction of deformable surgical scenes is a critical yet challenging task. Building on recent advancements in 3D Gaussian splatting, current methods have seen significant improvements in both reconstruction quality and rendering speed. However, two major limitations remain: (1) difficulty in handling irreversible dynamic changes, such as tissue shearing, which are common in surgical scenes; and (2) the lack of hierarchical modeling for surgical scene deformation, which reduces rendering speed. To address these challenges, we introduce EH-SurGS, an efficient and high-fidelity reconstruction algorithm for deformable surgical scenes. We propose a deformation modeling approach that incorporates the life cycle of 3D Gaussians, effectively capturing both regular and irreversible deformations, thus enhancing reconstruction quality. Additionally, we present an adaptive motion hierarchy strategy that distinguishes between static and deformable regions within the surgical scene. This strategy reduces the number of 3D Gaussians passing through the deformation field, thereby improving rendering speed. Extensive experiments demonstrate that our method surpasses existing state-of-the-art approaches in both reconstruction quality and rendering speed. Ablation studies further validate the effectiveness and necessity of our proposed components. We will open-source our code upon acceptance of the paper.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problem of efficient and high - fidelity reconstruction in deformable surgical scenes. Specifically, the authors point out that current methods have two main limitations in dealing with irreversible dynamic changes (such as tissue shearing) and the lack of hierarchical modeling:

1. **Handling of irreversible dynamic changes**: Irreversible dynamic changes (such as tissue shearing) commonly seen in surgical scenes are difficult to be effectively modeled by existing methods because there is no clear correspondence before and after these changes.
2. **Lack of hierarchical modeling**: Existing methods are unable to distinguish motions at different scales in surgical scenes, resulting in the need to transform even some static areas through deformation fields, thus reducing the rendering speed.

To solve these problems, the authors propose the EH - SurGS algorithm, and its main contributions are as follows:

1. **Introducing the concept of the life cycle of 3D Gaussian functions**: In order to simulate irreversible dynamic changes in surgical scenes, the authors introduce the concept of the life cycle of 3D Gaussian functions. This enables each 3D Gaussian function to be activated or disabled within a specific time period, thereby better capturing general and irreversible deformations and improving the reconstruction quality.

The life - cycle model is implemented by the following formula:
\[
b(t)=\exp\left(-\frac{1}{2\sigma^{2}}(t - \theta)^{2}\right)
\]
where \(\theta\) and \(\sigma\) are the center and variance of the Gaussian basis function respectively, and they are learnable parameters.

2. **Adaptive motion - level strategy**: To improve the rendering speed, the authors propose an adaptive motion - level strategy that can distinguish between static and deformable areas in surgical scenes. This significantly reduces the number of 3D Gaussian functions that need to be passed through the deformation field, thereby increasing the rendering speed.

This strategy updates the mask \(F\) by calculating the average deformation and dynamic/static rendering losses to distinguish between deformable and static areas. The specific steps include:
- Calculate the average deformation \(\Delta t\) of each area and the dynamic/static rendering losses \(L_d\), \(L_s\).
- Determine whether the area belongs to the potential dynamic area set \(Q\) or the static area set \(W\) according to the set thresholds \(\delta_1\) and \(\delta_2\).
- Dynamically update the mask \(F\) and adjust the update frequency \(N_m\) according to the optimization progress.

3. **Experimental verification**: Extensive experiments show that EH - SurGS outperforms existing state - of - the - art methods on multiple endoscope datasets and performs excellently in terms of reconstruction quality and rendering speed. Ablation studies further verify the effectiveness and necessity of the proposed components.

In summary, this paper solves the deficiencies of existing methods in dealing with irreversible dynamic changes and hierarchical modeling by introducing the concept of the life cycle of 3D Gaussian functions and the adaptive motion - level strategy, achieving more efficient and high - fidelity surgical scene reconstruction.