Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Framework

Jiandong Jin,Xiao Wang,Qian Zhu,Haiyang Wang,Chenglong Li

2024-08-19

Abstract:Pedestrian Attribute Recognition (PAR) is one of the indispensable tasks in human-centered research. However, existing datasets neglect different domains (e.g., environments, times, populations, and data sources), only conducting simple random splits, and the performance of these datasets has already approached saturation. In the past five years, no large-scale dataset has been opened to the public. To address this issue, this paper proposes a new large-scale, cross-domain pedestrian attribute recognition dataset to fill the data gap, termed MSP60K. It consists of 60,122 images and 57 attribute annotations across eight scenarios. Synthetic degradation is also conducted to further narrow the gap between the dataset and real-world challenging scenarios. To establish a more rigorous benchmark, we evaluate 17 representative PAR models under both random and cross-domain split protocols on our dataset. Additionally, we propose an innovative Large Language Model (LLM) augmented PAR framework, named LLM-PAR. This framework processes pedestrian images through a Vision Transformer (ViT) backbone to extract features and introduces a multi-embedding query Transformer to learn partial-aware features for attribute classification. Significantly, we enhance this framework with LLM for ensemble learning and visual feature augmentation. Comprehensive experiments across multiple PAR benchmark datasets have thoroughly validated the efficacy of our proposed framework. The dataset and source code accompanying this paper will be made publicly available at \url{<a class="link-external link-https" href="https://github.com/Event-AHU/OpenPAR" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper aims to address several key issues in the field of Pedestrian Attribute Recognition (PAR):

1. **Dataset Bottleneck**: The performance of existing PAR datasets has nearly reached saturation, and there have been almost no new large-scale datasets publicly released in the past 5 years. Therefore, there is an urgent need for a new benchmark dataset for pedestrian attribute recognition to support new research work.
2. **Neglect of Cross-Domain Influence**: Existing datasets only use random partitioning methods for model training and testing, ignoring the impact of different environments, times, crowds, and data sources on PAR models.
3. **Lack of Real-World Challenge Factors**: Current datasets fail to fully reflect the complex challenges in the real world (such as lighting changes, motion blur, etc.), which may lead to the neglect of data corruption impacts in practical applications, thereby introducing safety hazards.

To address the above issues, the authors propose a new large-scale cross-domain pedestrian attribute recognition dataset MSP60K, containing 60,122 images and 57 attribute annotations, covering eight scenes, and further simulating real-world complex situations through synthetic degradation techniques. Additionally, a PAR framework enhanced by large language models (LLM-PAR) is proposed, which combines visual feature extraction and text generation to improve attribute recognition performance. The effectiveness of the proposed method is validated through experiments on multiple benchmark datasets.