3D Gaussian Editing with A Single Image

Guan Luo,Tian-Xing Xu,Ying-Tian Liu,Xiao-Xiong Fan,Fang-Lue Zhang,Song-Hai Zhang

2024-08-14

Abstract:The modeling and manipulation of 3D scenes captured from the real world are pivotal in various applications, attracting growing research interest. While previous works on editing have achieved interesting results through manipulating 3D meshes, they often require accurately reconstructed meshes to perform editing, which limits their application in 3D content generation. To address this gap, we introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint of the original scene. To capture long-range object deformation, we introduce positional loss into the optimization process of 3D Gaussian Splatting and enable gradient propagation through reparameterization. To handle occluded 3D Gaussians when rendering from the specified viewpoint, we build an anchor-based structure and employ a coarse-to-fine optimization strategy capable of handling long-range deformation while maintaining structural stability. Furthermore, we design a novel masking strategy to adaptively identify non-rigid deformation regions for fine-scale modeling. Extensive experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation, demonstrating superior editing flexibility and quality compared to previous approaches.

Computer Vision and Pattern Recognition,Multimedia

What problem does this paper attempt to address?

The paper mainly addresses the following issues:

1. **Proposing a new 3D scene editing method**: This method is based on 3D Gaussian Splatting, using a single image as input to perform intuitive 3D scene editing, achieving a "what you see is what you get" editing effect.

2. **Handling long-range deformation issues**: To capture long-range object deformations and enable gradient propagation, a positional loss is introduced, and the entire process is made differentiable through reparameterization.

3. **Maintaining geometric consistency**: To address the issue of partial 3D Gaussians being occluded when rendering from specific viewpoints, an anchor-based as-rigid-as-possible (ARAP) regularization method and a coarse-to-fine optimization strategy are proposed to maintain object-level geometric stability.

4. **Adaptive identification of non-rigid deformation areas**: A novel masking strategy is designed to adaptively identify non-rigid deformation areas, thereby releasing ARAP regularization and achieving more precise geometric detail modeling.

5. **Overall contributions**:
- Proposed the first single-image-driven 3D Gaussian scene editing method, achieving a "what you see is what you get" editing approach.
- Introduced positional derivatives in 3D Gaussian Splatting to capture long-range deformations and support gradient propagation through reparameterization.
- Proposed an anchor-based as-rigid-as-possible regularization method and a coarse-to-fine optimization strategy to maintain object-level geometric consistency.
- Introduced an adaptive masking strategy to identify non-rigid deformation parts, ensuring more precise modeling.

Through the above methods, the paper aims to improve the efficiency and quality of 3D scene editing, especially in handling geometric details, long-range deformations, and non-rigid deformations. Experimental results demonstrate the effectiveness of this method, particularly in dealing with real-world data.