Abstract:Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.

Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model

Localized Gaussian Splatting Editing with Contextual Awareness

GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise

Robust 3D Gaussian Splatting for Novel View Synthesis in Presence of Distractors

SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis

Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation

Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting

Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors

EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting

GaussianPro: 3D Gaussian Splatting with Progressive Propagation

Diffusion Models with Anisotropic Gaussian Splatting for Image Inpainting

ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting

SplatFormer: Point Transformer for Robust 3D Gaussian Splatting

3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

GET3DGS: Generate 3D Gaussians Based on Points Deformation Fields

MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction

InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds