Abstract:Crowd understanding has aroused the widespread interest in vision domain due to its important practical significance. Unfortunately, there is no effort to explore crowd understanding in multi-modal domain that bridges natural language and computer vision. Referring expression comprehension (REF) is such a representative multi-modal task. Current REF studies focus more on grounding the target object from multiple distinctive categories in general scenarios. It is difficult to applied to complex real-world crowd understanding. To fill this gap, we propose a new challenging dataset, called RefCrowd, which towards looking for the target person in crowd with referring expressions. It not only requires to sufficiently mine the natural language information, but also requires to carefully focus on subtle differences between the target and a crowd of persons with similar appearance, so as to realize the fine-grained mapping from language to vision. Furthermore, we propose a Fine-grained Multi-modal Attribute Contrastive Network (FMAC) to deal with REF in crowd understanding. It first decomposes the intricate visual and language features into attribute-aware multi-modal features, and then captures discriminative but robustness fine-grained attribute features to effectively distinguish these subtle differences between similar persons. The proposed method outperforms existing state-of-the-art (SoTA) methods on our RefCrowd dataset and existing REF datasets. In addition, we implement an end-to-end REF toolbox for the deeper research in multi-modal domain. Our dataset and code can be available at: \url{<a class="link-external link-https" href="https://qiuheqian.github.io/datasets/refcrowd/" rel="external noopener nofollow">this https URL</a>}.

HumanFormer: Human-centric Prompting Multi-modal Perception Transformer for Referring Crowd Detection

RefCrowd: Grounding the Target in Crowd with Referring Expressions

See Extensively While Focusing on the Core Area for Pedestrian Detection.

RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios

Referring Human Pose and Mask Estimation in the Wild

Attribute-Prompting Multi-Modal Object Reasoning Transformer for Remote Sensing Visual Grounding

Adaptive multimodal prompt for human-object interaction with local feature enhanced transformer

RAG-based Crowdsourcing Task Decomposition via Masked Contrastive Learning with Prompts

Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection

Reference Prompted Model Adaptation for Referring Camouflaged Object Detection

TemPrompt: Multi-Task Prompt Learning for Temporal Relation Extraction in RAG-based Crowdsourcing Systems

An interactive network based on transformer for multimodal crowd counting

ProFD: Prompt-Guided Feature Disentangling for Occluded Person Re-Identification

A Novel Part Refinement Tandem Transformer for Human-Object Interaction Detection

OBhunter: An ensemble spectral-angular based transformer network for occlusion detection

TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding

Double Anchor R-CNN for Human Detection in a Crowd

Multi-modal Crowd Counting via Modal Emulation

PS-RCNN: Detecting Secondary Human Instances in a Crowd via Primary Object Suppression

Hierarchical Inverse Distance Transformer for Enhanced Localization in Dense Crowds

Exploring a Distillation with Embedded Prompts for Object Detection in Adverse Environments.