Abstract:Text-to-3D form plays a crucial role in creating editable 3D scenes for AR/VR. Recent advances have shown promise in merging neural radiance fields (NeRFs) with pre-trained diffusion models for text-to-3D object generation. However, one enduring challenge is their inadequate capability to accurately parse and regenerate consistent multi-object environments. Specifically, these models encounter difficulties in accurately representing quantity and style prompted by multi-object texts, often resulting in a collapse of the rendering fidelity that fails to match the semantic intricacies. Moreover, amalgamating these elements into a coherent 3D scene is a substantial challenge, stemming from generic distribution inherent in diffusion models. To tackle the issue of 'guidance collapse' and further enhance scene consistency, we propose a novel framework, dubbed CompoNeRF, by integrating an editable 3D scene layout with object-specific and scene-wide guidance mechanisms. It initiates by interpreting a complex text into the layout populated with multiple NeRFs, each paired with a corresponding subtext prompt for precise object depiction. Next, a tailored composition module seamlessly blends these NeRFs, promoting consistency, while the dual-level text guidance reduces ambiguity and boosts accuracy. Noticeably, our composition design permits decomposition. This enables flexible scene editing and recomposition into new scenes based on the edited layout or text prompts. Utilizing the open-source Stable Diffusion model, CompoNeRF generates multi-object scenes with high fidelity. Remarkably, our framework achieves up to a \textbf{54\%} improvement by the multi-view CLIP score metric. Our user study indicates that our method has significantly improved semantic accuracy, multi-view consistency, and individual recognizability for multi-object scene generation.

Blending-NeRF: Text-Driven Localized Editing in Neural Radiance Fields

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF

RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout

ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context

Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model

FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields

MorphNeRF: Text-Guided 3D-aware Editing via Morphing Generative Neural Radiance Fields

NeRF-Insert: 3D Local Editing with Multimodal Control Signals

NeRF-Art: Text-Driven Neural Radiance Fields Stylization

Local 3D Editing via 3D Distillation of CLIP Knowledge

DATENeRF: Depth-Aware Text-based Editing of NeRFs

4D-Editor: Interactive Object-level Editing in Dynamic Neural Radiance Fields via Semantic Distillation

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

SIn-NeRF2NeRF: Editing 3D Scenes with Instructions through Segmentation and Inpainting

3D-aware Blending with Generative NeRFs

Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields

Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation

BluNF: Blueprint Neural Field