Abstract:Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.

Sketch-2-4d: Sketch Driven Dynamic 3d Scene Generation

Sketch Based Multi-source 3D Animation Transfer

Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches

PENCIL: A Framework for Expressing Free-Hand Sketching in 3D

SketchDream: Sketch-based Text-To-3D Generation and Editing

PaintScene4D: Consistent 4D Scene Generation from Text Prompts

Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models

Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Comp4D: LLM-Guided Compositional 4D Scene Generation

DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos

Deep3DSketch+\+: High-Fidelity 3D Modeling from Single Free-hand Sketches

Rapid 3D Model Generation with Intuitive 3D Input

External Knowledge Enhanced 3D Scene Generation from Sketch

Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation

4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency

Semantic Score Distillation Sampling for Compositional Text-to-3D Generation

EG4D: Explicit Generation of 4D Object without Score Distillation

SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches