FullFormer: Generating Shapes Inside Shapes

Tejaswini Medi,Jawad Tayyub,Muhammad Sarmad,Frank Lindseth,Margret Keuper
2023-03-21
Abstract:Implicit generative models have been widely employed to model 3D data and have recently proven to be successful in encoding and generating high-quality 3D shapes. This work builds upon these models and alleviates current limitations by presenting the first implicit generative model that facilitates the generation of complex 3D shapes with rich internal geometric details. To achieve this, our model uses unsigned distance fields to represent nested 3D surfaces allowing learning from non-watertight mesh data. We propose a transformer-based autoregressive model for 3D shape generation that leverages context-rich tokens from vector quantized shape embeddings. The generated tokens are decoded into an unsigned distance field which is rendered into a novel 3D shape exhibiting a rich internal structure. We demonstrate that our model achieves state-of-the-art point cloud generation results on popular classes of 'Cars', 'Planes', and 'Chairs' of the ShapeNet dataset. Additionally, we curate a dataset that exclusively comprises shapes with realistic internal details from the `Cars' class of ShapeNet and demonstrate our method's efficacy in generating these shapes with internal geometry.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper aims to address the problem of generating complex 3D shapes with rich internal geometric details. Specifically, the research team proposes a new generative framework called FullFormer, which leverages Unsigned Distance Fields (UDFs) to represent nested 3D surfaces and generates these complex 3D shapes through an autoregressive Transformer-based model. The main contributions of FullFormer include: 1. Proposing a neural network-based implicit generative framework for generating 3D shapes with internal details. 2. The proposed model can learn from non-watertight 3D data, expanding the types of applicable data. 3. The researchers have meticulously curated a dataset of car shapes with real internal details and made it publicly available. 4. Experimental results show that the method achieves state-of-the-art quantitative and qualitative results on the point cloud generation tasks for the "car," "airplane," and "chair" categories in the popular ShapeNet dataset. In short, FullFormer combines the advantages of UDFs representation and the capabilities of Transformer models to generate high-quality and diverse 3D shapes with complex internal structures, such as car seats. This approach is significant for fields like product design, robotics, CAD design, and medical imaging, as applications in these areas often require precise modeling and simulation of internal geometric structures.