PocoLoco: A Point Cloud Diffusion Model of Human Shape in Loose Clothing

Siddharth Seth,Rishabh Dabral,Diogo Luvizon,Marc Habermann,Ming-Hsuan Yang,Christian Theobalt,Adam Kortylewski

2024-11-07

Abstract:Modeling a human avatar that can plausibly deform to articulations is an active area of research. We present PocoLoco -- the first template-free, point-based, pose-conditioned generative model for 3D humans in loose clothing. We motivate our work by noting that most methods require a parametric model of the human body to ground pose-dependent deformations. Consequently, they are restricted to modeling clothing that is topologically similar to the naked body and do not extend well to loose clothing. The few methods that attempt to model loose clothing typically require either canonicalization or a UV-parameterization and need to address the challenging problem of explicitly estimating correspondences for the deforming clothes. In this work, we formulate avatar clothing deformation as a conditional point-cloud generation task within the denoising diffusion framework. Crucially, our framework operates directly on unordered point clouds, eliminating the need for a parametric model or a clothing template. This also enables a variety of practical applications, such as point-cloud completion and pose-based editing -- important features for virtual human animation. As current datasets for human avatars in loose clothing are far too small for training diffusion models, we release a dataset of two subjects performing various poses in loose clothing with a total of 75K point clouds. By contributing towards tackling the challenging task of effectively modeling loose clothing and expanding the available data for training these models, we aim to set the stage for further innovation in digital humans. The source code is available at <a class="link-external link-https" href="https://github.com/sidsunny/pocoloco" rel="external noopener nofollow">this https URL</a> .

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in human body modeling with loose clothing, existing methods usually rely on parametric human body models or clothing templates, which limits their performance when dealing with loose clothing whose topological structure is quite different from the naked body shape. Specifically, when simulating loose clothing, existing methods often produce artifacts such as surface tearing or wrinkling, and require template fitting for input scans, which is a cumbersome process and may not meet the needs of artists.

To address these problems, the paper proposes PocoLoco - the first template - free, point - cloud - based conditional generation model for 3D human body modeling with loose clothing. PocoLoco operates directly on unordered point clouds without assuming point correspondences, thus avoiding the need to use parametric models or clothing templates. This method can not only simulate the deformation of loose clothing more naturally, but also supports functions such as point cloud completion and pose - based editing, which are very important for the creation of virtual human animations.

In addition, since the current scale of human body data sets with loose clothing is too small to train diffusion models, the author also releases a high - quality 3D reconstruction data set containing two subjects wearing loose clothing in different postures, with a total of 75,000 point clouds. By contributing more data, the author aims to lay the foundation for future innovations in the field of digital humans.