An Empirical Study of Mamba-based Pedestrian Attribute Recognition

Xiao Wang,Weizhe Kong,Jiandong Jin,Shiao Wang,Ruichong Gao,Qingchuan Ma,Chenglong Li,Jin Tang

2024-07-15

Abstract:Current strong pedestrian attribute recognition models are developed based on Transformer networks, which are computationally heavy. Recently proposed models with linear complexity (e.g., Mamba) have garnered significant attention and have achieved a good balance between accuracy and computational cost across a variety of visual tasks. Relevant review articles also suggest that while these models can perform well on some pedestrian attribute recognition datasets, they are generally weaker than the corresponding Transformer models. To further tap into the potential of the novel Mamba architecture for PAR tasks, this paper designs and adapts Mamba into two typical PAR frameworks, i.e., the text-image fusion approach and pure vision Mamba multi-label recognition framework. It is found that interacting with attribute tags as additional input does not always lead to an improvement, specifically, Vim can be enhanced, but VMamba cannot. This paper further designs various hybrid Mamba-Transformer variants and conducts thorough experimental validations. These experimental results indicate that simply enhancing Mamba with a Transformer does not always lead to performance improvements but yields better results under certain settings. We hope this empirical study can further inspire research in Mamba for PAR, and even extend into the domain of multi-label recognition, through the design of these network structures and comprehensive experimentation. The source code of this work will be released at \url{<a class="link-external link-https" href="https://github.com/Event-AHU/OpenPAR" rel="external noopener nofollow">this https URL</a>}

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve several key problems existing in current models in the Pedestrian Attribute Recognition (PAR) task:

1. **High computational cost**: Although existing Transformer - based pedestrian attribute recognition models perform well in terms of accuracy, due to the self - attention mechanism, their computational complexity is \(O(N^2)\), resulting in high training and inference costs, especially when applied to large - scale datasets.

2. **Limited performance in challenging scenarios**: Although existing PAR models perform well in simple scenarios, their recognition performance is still limited in challenging scenarios such as low - light and cross - domain.

3. **Explore the potential of linear - complexity models**: Recently proposed linear - complexity models (such as Mamba) have achieved a good balance in multiple visual tasks, maintaining high accuracy while significantly reducing computational costs. However, how these models perform in the pedestrian attribute recognition task and how to further optimize their performance remain a research - worthy problem.

To this end, this paper designs and adapts the Mamba architecture into two typical PAR frameworks, namely the text - image fusion method and the pure - visual Mamba multi - label recognition framework. Through experimental verification, the paper finds that interacting with attribute labels as additional inputs does not always lead to performance improvement, specifically, Vim can be enhanced, but VMamba cannot. In addition, the paper also designs a variety of hybrid Mamba - Transformer variants and conducts thorough experimental verification. The experimental results show that simply enhancing the Mamba model does not always improve performance, but better results can be achieved in some settings.

Overall, through this empirical study, this paper hopes to further stimulate research on Mamba in PAR tasks and even expand to the multi - label recognition field by designing these network structures and comprehensive experiments.