SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation

Zhongang Cai,Wanqi Yin,Ailing Zeng,Chen Wei,Qingping Sun,Yanjun Wang,Hui En Pang,Haiyi Mei,Mingyuan Zhang,Lei Zhang,Chen Change Loy,Lei Yang,Ziwei Liu

2024-07-28

Abstract:Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods still depend largely on a confined set of training datasets. In this work, we investigate scaling up EHPS towards the first generalist foundation model (dubbed SMPLer-X), with up to ViT-Huge as the backbone and training with up to 4.5M instances from diverse data sources. With big data and the large model, SMPLer-X exhibits strong performance across diverse test benchmarks and excellent transferability to even unseen environments. 1) For the data scaling, we perform a systematic investigation on 32 EHPS datasets, including a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. 2) For the model scaling, we take advantage of vision transformers to study the scaling law of model sizes in EHPS. Moreover, our finetuning strategy turn SMPLer-X into specialist models, allowing them to achieve further performance boosts. Notably, our foundation model SMPLer-X consistently delivers state-of-the-art results on seven benchmarks such as AGORA (107.2 mm NMVE), UBody (57.4 mm PVE), EgoBody (63.6 mm PVE), and EHF (62.3 mm PVE without finetuning). Homepage: <a class="link-external link-https" href="https://caizhongang.github.io/projects/SMPLer-X/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that current expressive human pose and shape estimation (EHPS) methods have insufficient generalization ability for diverse scenarios. Although existing EHPS methods have made significant progress on specific datasets, these methods still rely heavily on limited datasets, which restricts their performance in unseen environments. Therefore, the goal of the paper is to establish a general - purpose base model (called SMPLer - X) by increasing the amount of training data and the model scale, in order to improve the performance of EHPS methods on different test benchmarks and enhance their cross - domain transfer ability.

Specifically, the paper focuses on the following aspects:
1. **Data Expansion**: Systematically studied 32 EHPS datasets, covering a wide range of application scenarios. By optimizing the training scheme and selecting appropriate datasets, the paper achieved a significant improvement in EHPS capabilities.
2. **Model Expansion**: Utilized Vision Transformers to study the expansion law of model size in EHPS. In addition, through the fine - tuning strategy, the base model was transformed into a professional model, further improving the performance.
3. **Benchmark Testing**: Established the first systematic EHPS benchmark test, evaluated their performance on five major benchmarks using 32 datasets, revealed significant inconsistencies between different benchmarks, and emphasized the need to use a large number of datasets to overcome the domain gap between different scenarios.

Through these efforts, SMPLer - X has excellent performance in multiple benchmarks, for example, achieving state - of - the - art performance on datasets such as AGORA, UBody, EgoBody, and EHF.