Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models

Zhengming Yu,Zhiyang Dou,Xiaoxiao Long,Cheng Lin,Zekun Li,Yuan Liu,Norman Müller,Taku Komura,Marc Habermann,Christian Theobalt,Xin Li,Wenping Wang

2024-07-24

Abstract:We present Surf-D, a novel method for generating high-quality 3D shapes as Surfaces with arbitrary topologies using Diffusion models. Previous methods explored shape generation with different representations and they suffer from limited topologies and poor geometry details. To generate high-quality surfaces of arbitrary topologies, we use the Unsigned Distance Field (UDF) as our surface representation to accommodate arbitrary topologies. Furthermore, we propose a new pipeline that employs a point-based AutoEncoder to learn a compact and continuous latent space for accurately encoding UDF and support high-resolution mesh extraction. We further show that our new pipeline significantly outperforms the prior approaches to learning the distance fields, such as the grid-based AutoEncoder, which is not scalable and incapable of learning accurate UDF. In addition, we adopt a curriculum learning strategy to efficiently embed various surfaces. With the pretrained shape latent space, we employ a latent diffusion model to acquire the distribution of various shapes. Extensive experiments are presented on using Surf-D for unconditional generation, category conditional generation, image conditional generation, and text-to-shape tasks. The experiments demonstrate the superior performance of Surf-D in shape generation across multiple modalities as conditions. Visit our project page at <a class="link-external link-https" href="https://yzmblog.github.io/projects/SurfD/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper aims to address the challenges encountered in 3D shape generation, particularly in generating high-quality 3D shapes with arbitrary topologies and high geometric details. Existing methods are often limited by restricted topologies and insufficient geometric details in shape generation. To overcome these challenges, the authors propose Surf-D, a novel approach based on a diffusion model that uses the Unsigned Distance Field (UDF) as a surface representation to generate high-quality 3D shapes with arbitrary topologies.

Specifically, the main contributions of Surf-D include:

1. **Proposing a new surface generation model**: This is the first model based on a diffusion model that uses UDF to generate 3D shapes with arbitrary topologies and detailed geometric features.
2. **Designing a point-based autoencoder**: This encoder can effectively query the gradient of a given point, enabling high-resolution and detail-preserving surface generation. To enhance surface learning, a curriculum learning strategy is also implemented.
3. **Excelling in shape generation tasks under various modalities**: Surf-D can generate 3D shapes with diverse topologies and handle generation tasks under category conditions, sketch conditions, single-view reconstruction, and text guidance.

Through these innovations, experimental results on multiple datasets demonstrate that Surf-D outperforms existing methods in shape generation, 3D reconstruction, and text-to-shape tasks, showcasing its superior performance in generating high-quality 3D shapes.