Abstract:Given a natural language expression and an image/video, the goal of referring segmentation is to produce the pixel-level masks of the entities described by the subject of the expression. Previous approaches tackle this problem by implicit feature interaction and fusion between visual and linguistic modalities in a one-stage manner. However, human tends to solve the referring problem in a progressive manner based on informative words in the expression, i.e., first roughly locating candidate entities and then distinguishing the target one. In this paper, we propose a cross-modal progressive comprehension (CMPC) scheme to effectively mimic human behaviors and implement it as a CMPC-I (Image) module and a CMPC-V (Video) module to improve referring image and video segmentation models. For image data, our CMPC-I module first employs entity and attribute words to perceive all the related entities that might be considered by the expression. Then, the relational words are adopted to highlight the target entity as well as suppress other irrelevant ones by spatial graph reasoning. For video data, our CMPC-V module further exploits action words based on CMPC-I to highlight the correct entity matched with the action cues by temporal graph reasoning. In addition to the CMPC, we also introduce a simple yet effective Text-Guided Feature Exchange (TGFE) module to integrate the reasoned multimodal features corresponding to different levels in the visual backbone under the guidance of textual information. In this way, multi-level features can communicate with each other and be mutually refined based on the textual context. Combining CMPC-I or CMPC-V with TGFE can form our image or video version referring segmentation frameworks and our frameworks achieve new state-of-the-art performances on four referring image segmentation benchmarks and three referring video segmentation benchmarks respectively. Our code is available at https://github.com/spyflying/CMPC-Refseg.

RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios

Referring Human Pose and Mask Estimation in the Wild

UniHCP: A Unified Model for Human-Centric Perceptions

HumanFormer: Human-centric Prompting Multi-modal Perception Transformer for Referring Crowd Detection

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

RefMask3D: Language-Guided Transformer for 3D Referring Segmentation

RefCrowd: Grounding the Target in Crowd with Referring Expressions

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

A Hybrid Framework for Referring Image Segmentation: Dual-Decoder Model with SAM Complementation

HSMR: A Head-Shoulder Mask Aided ResNet to Guide Focus of Re-Identification Implemented on Tour-Guide Robot.

Rethinking referring relationships from a perspective of mask-level relational reasoning

Dual-graph Hierarchical Interaction Network for Referring Image Segmentation

YouRefIt: Embodied Reference Understanding with Language and Gesture

Referring Image Segmentation via Cross-Modal Progressive Comprehension

Human-Centric Image Captioning

CMIRNet: Cross-Modal Interactive Reasoning Network for Referring Image Segmentation

Cross-Modal Progressive Comprehension for Referring Segmentation

Towards Unifying Reference Expression Generation and Comprehension

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems