Adept: Annotation-denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining
Weizhen He,Yunfeng Yan,Shixiang Tang,Yiheng Deng,Yangyang Zhong,Pengxin Luo,Donglian Qi
DOI: https://doi.org/10.1016/j.neucom.2024.128694
IF: 6
2025-01-01
Neurocomputing
Abstract:Human-centric perception is the core of diverse computer vision tasks and has been a long-standing research focus. However, previous research studied these human-centric tasks individually, whose performance is largely limited to the size of the public task-specific datasets. Recent human-centric methods leverage the additional modalities, e.g., depth, to learn fine-grained semantic information, which limits the benefit of pretraining models due to their sensitivity to camera views and the scarcity of RGB-D data on the Internet. This paper improves the data scalability of human-centric pretraining methods by discarding depth information and exploring semantic information of RGB images in the frequency space by Discrete Cosine Transform (DCT). We further propose new annotation denoising auxiliary tasks with keypoints and DCT maps to enforce the RGB image extractor to learn fine-grained semantic information of human bodies. Our extensive experiments show that when pretrained on large-scale datasets (COCO and AIC datasets) without depth annotation, our model achieves better performance than state-of-the-art methods by +0.5 mAP (]) on COCO, +1.4 PCKh (]) on MPII and-0.51 EPE (down arrow) on Human3.6M for pose estimation, by +4.50 mIoU (]) on Human3.6M for human parsing, by-3.14 MAE (down arrow) on SHA and-0.07 MAE (down arrow) on SHB for crowd counting, by +1.1 F1 score(]) on SHA and +0.8 F1 score(]) on SHA for crowd localization, and by +0.1 mAP (]) on Market1501 and +0.8 mAP (]) on MSMT for person ReID. We also validate the effectiveness of our method on MPII+NTURGBD datasets.