PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations

Yang Zheng,Qingqing Zhao,Guandao Yang,Wang Yifan,Donglai Xiang,Florian Dubost,Dmitry Lagun,Thabo Beeler,Federico Tombari,Leonidas Guibas,Gordon Wetzstein

2024-04-09

Abstract:Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at:

Graphics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper proposes a new framework called PhysAvatar to address the challenges in reconstructing 3D virtual avatars wearing clothing from multi-view video data. Existing methods struggle to accurately simulate the dynamic behavior of loose-fitting garments when reconstructing the human body. PhysAvatar combines inverse rendering with inverse physics to automatically estimate the physical parameters of the human body shape, appearance, and clothing material. By using spatio-temporal mesh tracking and physics-based inverse rendering, the intrinsic material properties of the clothing are estimated. PhysAvatar includes a physics simulator that estimates the physical parameters of the clothing in a systematic manner using gradient optimization. This approach enables PhysAvatar to generate high-quality new view renderings under unseen actions and lighting conditions in the training data, leading to significant advancements in creating realistic digital humans.

The key contributions of the paper are as follows:
1. Proposed a new 3D virtual avatar inverse rendering paradigm that systematically incorporates the physical characteristics of loose-fitting garments.
2. A comprehensive pipeline that includes accurate and efficient mesh reconstruction and tracking, automatic optimization of clothing physical material, and physics-based inverse rendering.

PhysAvatar solves the problem of disregarding dynamic modeling of loose-fitting garments in existing methods and is capable of handling the dynamic behavior of clothing under unseen motion and lighting conditions, thus improving the realism of rendering. Compared to other methods, PhysAvatar accurately recovers shape and re-illuminatable appearance through physics-based inverse rendering and optimization, achieving high-fidelity results.