Abstract:3D object generation has undergone significant advancements, yielding high-quality results. However, fall short of achieving precise user control, often yielding results that do not align with user expectations, thus limiting their applicability. User-envisioning 3D object generation faces significant challenges in realizing its concepts using current generative models due to limited interaction capabilities. Existing methods mainly offer two approaches: (i) interpreting textual instructions with constrained controllability, or (ii) reconstructing 3D objects from 2D images. Both of them limit customization to the confines of the 2D reference and potentially introduce undesirable artifacts during the 3D lifting process, restricting the scope for direct and versatile 3D modifications. In this work, we introduce Interactive3D, an innovative framework for interactive 3D generation that grants users precise control over the generative process through extensive 3D interaction capabilities. Interactive3D is constructed in two cascading stages, utilizing distinct 3D representations. The first stage employs Gaussian Splatting for direct user interaction, allowing modifications and guidance of the generative direction at any intermediate step through (i) Adding and Removing components, (ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv) Semantic Editing. Subsequently, the Gaussian splats are transformed into InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to further add details and extract the geometry in the second stage. Our experiments demonstrate that Interactive3D markedly improves the controllability and quality of 3D generation. Our project webpage is available at \url{

3Description: An Intuitive Human-AI Collaborative 3D Modeling Approach

Using the CAT for 3D Sketching in Front of Large Displays

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

ProtoDreamer: A Mixed-prototype Tool Combining Physical Model and Generative AI to Support Conceptual Design

Rapid 3D Model Generation with Intuitive 3D Input

High-Fidelity 3D Face Generation from Natural Language Descriptions

New model of 3D collaborative awareness and its application

Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only

Implementation of Artificial Intelligence Based Image Creation Technology for Conceptual Ideas in 3D Visual Modeling

Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

A Framework to Study Human-AI Collaborative Design Space Exploration

Deconstructing Human-AI Collaboration: Agency, Interaction, and Adaptation

"Hey Model!" - Natural User Interactions and Agency in Accessible Interactive 3D Models

A New Model of 3D Collaborative Awareness and Its Application Reserach

Interactive3D: Create What You Want by Interactive 3D Generation

Digital Life Project: Autonomous 3D Characters with Social Intelligence

iControl3D: An Interactive System for Controllable 3D Scene Generation

Make-A-Character: High Quality Text-to-3D Character Generation within Minutes

TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions

ChatHuman: Language-driven 3D Human Understanding with Retrieval-Augmented Tool Reasoning