Abstract:Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.

Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing

Efficient Density Control for 3D Gaussian Splatting

GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling

GeoTexDensifier: Geometry-Texture-Aware Densification for High-Quality Photorealistic 3D Gaussian Splatting

Textured-GS: Gaussian Splatting with Spatially Defined Color and Opacity

HDGS: Textured 2D Gaussian Splatting for Enhanced Scene Rendering

3D-HGS: 3D Half-Gaussian Splatting

AbsGS: Recovering Fine Details for 3D Gaussian Splatting

2DGH: 2D Gaussian-Hermite Splatting for High-quality Rendering and Better Geometry Reconstruction

GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal

3D Gaussian Model for Animation and Texturing

Spectral-GS: Taming 3D Gaussian Splatting with Spectral Entropy

GeoGaussian: Geometry-aware Gaussian Splatting for Scene Rendering

2D Gaussian Splatting for Geometrically Accurate Radiance Fields

A Survey on 3D Gaussian Splatting

Localized Gaussian Splatting Editing with Contextual Awareness

Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition

MVG-Splatting: Multi-View Guided Gaussian Splatting with Adaptive Quantile-Based Geometric Consistency Densification

LineGS : 3D Line Segment Representation on 3D Gaussian Splatting