Abstract:Generating diverse and high-quality 3D assets automatically poses a fundamental yet challenging task in 3D computer vision. Despite extensive efforts in 3D generation, existing optimization-based approaches struggle to produce large-scale 3D assets efficiently. Meanwhile, feed-forward methods often focus on generating only a single category or a few categories, limiting their generalizability. Therefore, we introduce a diffusion-based feed-forward framework to address these challenges with a single model. To handle the large diversity and complexity in geometry and texture across categories efficiently, we 1) adopt improved triplane to guarantee efficiency; 2) introduce the 3D-aware transformer to aggregate the generalized 3D knowledge with specialized 3D features; and 3) devise the 3D-aware encoder/decoder to enhance the generalized 3D knowledge. Building upon our 3D-aware Diffusion model with TransFormer, DiffTF, we propose a stronger version for 3D generation, i.e., DiffTF++. It boils down to two parts: multi-view reconstruction loss and triplane refinement. Specifically, we utilize multi-view reconstruction loss to fine-tune the diffusion model and triplane decoder, thereby avoiding the negative influence caused by reconstruction errors and improving texture synthesis. By eliminating the mismatch between the two stages, the generative performance is enhanced, especially in texture. Additionally, a 3D-aware refinement process is introduced to filter out artifacts and refine triplanes, resulting in the generation of more intricate and reasonable details. Extensive experiments on ShapeNet and OmniObject3D convincingly demonstrate the effectiveness of our proposed modules and the state-of-the-art 3D object generation performance with large diversity, rich semantics, and high quality.

TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation

TD3D: Tensor-based Discrete Diffusion Process for 3D Shape Generation

Deformable 3D Shape Diffusion Model

PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models

Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation

RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation

PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models

OctFusion: Octree-based Diffusion Models for 3D Shape Generation

3D MedDiffusion: A 3D Medical Diffusion Model for Controllable and High-quality Medical Image Generation

GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

HOLODIFFUSION: Training a 3D Diffusion Model using 2D Images

Diffusion Models for Memory-efficient Processing of 3D Medical Images

3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

Generating Images with 3D Annotations Using Diffusion Models

SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation

DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generation

Locally Attentional SDF Diffusion for Controllable 3D Shape Generation

Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features

Memory-Efficient 3D Denoising Diffusion Models for Medical Image Processing

3D Shape Generation and Completion through Point-Voxel Diffusion