DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose

Yusuke Yoshiyasu,Leyuan Sun

2024-08-27

Abstract:This paper presents DiffSurf, a transformer-based denoising diffusion model for generating and reconstructing 3D surfaces. Specifically, we design a diffusion transformer architecture that predicts noise from noisy 3D surface vertices and normals. With this architecture, DiffSurf is able to generate 3D surfaces in various poses and shapes, such as human bodies, hands, animals and man-made objects. Further, DiffSurf is versatile in that it can address various 3D downstream tasks including morphing, body shape variation and 3D human mesh fitting to 2D keypoints. Experimental results on 3D human model benchmarks demonstrate that DiffSurf can generate shapes with greater diversity and higher quality than previous generative models. Furthermore, when applied to the task of single-image 3D human mesh recovery, DiffSurf achieves accuracy comparable to prior techniques at a near real-time rate.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Generating diverse poses and shapes of 3D surfaces**: Existing diffusion models lack the ability to maintain point-to-point correspondence when generating different poses of 3D human or animal bodies, leading to insufficient quality and diversity in the generated results. DiffSurf ensures point-to-point correspondence between different shapes by introducing a vertex-based mesh recovery paradigm.

2. **Handling various object types**: Researchers want the model to handle a wide range of object types, such as human bodies, mammals, and artificial objects. DiffSurf is designed with a general network architecture that can generate and reconstruct 3D surfaces in various shapes and poses.

3. **Multifunctional framework**: Researchers need a framework capable of handling multiple tasks (e.g., interpolating between two shapes, changing poses, and manipulating shapes). DiffSurf provides a method based on pre-trained models that can effectively utilize these models to solve various 3D processing tasks.

By proposing DiffSurf—a Transformer-based denoising diffusion model, the paper addresses the above challenges, achieving high-quality and diverse 3D surface generation, and performs well in different downstream tasks.