Abstract:Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space. In this paper, we extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously. Firstly, we leverage an ensemble of publicly available 3D datasets to facilitate the training of large-scale models. It consists of a comprehensive collection of approximately 900,000 objects, with multiple properties of meshes, points, voxels, rendered images, and text captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our model to learn from a wide range of object variations. However, directly applying 3D auto-regression encounters critical challenges of high computational demands on volumetric grids and ambiguous auto-regressive order along grid dimensions, resulting in inferior quality of 3D shapes. To this end, we then present a novel framework Argus3D in terms of capacity. Concretely, our approach introduces discrete representation learning based on a latent vector instead of volumetric grids, which not only reduces computational costs but also preserves essential geometric details by learning the joint distributions in a more tractable order. The capacity of conditional generation can thus be realized by simply concatenating various conditioning inputs to the latent vector, such as point clouds, categories, images, and texts. In addition, thanks to the simplicity of our model architecture, we naturally scale up our approach to a larger model with an impressive 3.6 billion parameters, further enhancing the quality of versatile 3D generation. Extensive experiments on four generation tasks demonstrate that Argus3D can synthesize diverse and faithful shapes across multiple categories, achieving remarkable performance.

EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation

MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers

Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale

$R^2$-Mesh: Reinforcement Learning Powered Mesh Reconstruction via Geometry and Appearance Refinement

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels

Learning Versatile 3D Shape Generation with Improved AR Models

High-Fidelity 3D Model Generation with Relightable Appearance from Single Freehand Sketches and Text Guidance

MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenization

MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

MeshingNet: A New Mesh Generation Method based on Deep Learning

PolyGen: An Autoregressive Generative Model of 3D Meshes

GetMesh: A Controllable Model for High-quality Mesh Generation and Manipulation

Learning Versatile 3D Shape Generation with Improved Auto-regressive Models.

Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability

Neural Progressive Meshes

3DGen: Triplane Latent Diffusion for Textured Mesh Generation

MeshArt: Generating Articulated Meshes with Structure-guided Transformers

SRL-Assisted AFM: Generating Planar Unstructured Quadrilateral Meshes with Supervised and Reinforcement Learning-Assisted Advancing Front Method

Reinforcement Learning for Automatic Quadrilateral Mesh Generation: a Soft Actor-Critic Approach

Neural Volumetric Mesh Generator