GAST: Sequential Gaussian Avatars with Hierarchical Spatio-temporal Context

Wangze Xu,Yifan Zhan,Zhihang Zhong,Xiao Sun
2024-11-25
Abstract:3D human avatars, through the use of canonical radiance fields and per-frame observed warping, enable high-fidelity rendering and animating. However, existing methods, which rely on either spatial SMPL(-X) poses or temporal embeddings, respectively suffer from coarse rendering quality or limited animation flexibility. To address these challenges, we propose GAST, a framework that unifies 3D human modeling with 3DGS by hierarchically integrating both spatial and temporal information. Specifically, we design a sequential conditioning framework for the non-rigid warping of the human body, under whose guidance more accurate 3D Gaussians can be obtained in the observation space. Moreover, the explicit properties of Gaussians allow us to embed richer sequential information, encompassing both the coarse sequence of human poses and finer per-vertex motion details. These sequence conditions are further sampled across different temporal scales, in a coarse-to-fine manner, ensuring unbiased inputs for non-rigid warping. Experimental results demonstrate that our method combined with hierarchical spatio-temporal modeling surpasses concurrent baselines, delivering both high-quality rendering and flexible animating capabilities.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to achieve high - fidelity rendering and flexible animation simultaneously in 3D human modeling. Existing methods either rely on spatially SMPL(-X) poses, resulting in rough rendering quality; or rely on time embedding, which has high rendering quality but limited animation flexibility. These problems mainly stem from the insufficiency of non - rigid deformation modeling and require a trade - off between rendering quality and animation capabilities. To overcome these challenges, the paper proposes the GAST framework to unify 3D human modeling with 3D Gaussian representation (3DGS) by hierarchically integrating spatial and temporal information. Specifically, GAST designs a sequential conditional framework for modeling non - rigid deformation, under which the 3D Gaussian distribution in the observation space can be obtained more accurately. In addition, by using the explicit characteristics of the Gaussian distribution, richer sequence information can be embedded, including coarse human pose sequences and finer - grained vertex motion details. These sequence conditions are sampled at different time scales to ensure unbiased input for non - rigid deformation. Experimental results show that the method combining hierarchical spatio - temporal modeling outperforms existing baseline methods in both rendering quality and animation flexibility.