Collaborative Position Reasoning Network for Referring Image Segmentation

Jianjian Cao,Beiya Dai,Yulin Li,Xiameng Qin,Jingdong Wang

2024-01-22

Abstract:Given an image and a natural language expression as input, the goal of referring image segmentation is to segment the foreground masks of the entities referred by the expression. Existing methods mainly focus on interactive learning between vision and language to enhance the multi-modal representations for global context reasoning. However, predicting directly in pixel-level space can lead to collapsed positioning and poor segmentation results. Its main challenge lies in how to explicitly model entity localization, especially for non-salient entities. In this paper, we tackle this problem by executing a Collaborative Position Reasoning Network (CPRN) via the proposed novel Row-and-Column interactive (RoCo) and Guided Holistic interactive (Holi) modules. Specifically, RoCo aggregates the visual features into the row- and column-wise features corresponding two directional axes respectively. It offers a fine-grained matching behavior that perceives the associations between the linguistic features and two decoupled visual features to perform position reasoning over a hierarchical space. Holi integrates features of the two modalities by a cross-modal attention mechanism, which suppresses the irrelevant redundancy under the guide of positioning information from RoCo. Thus, with the incorporation of RoCo and Holi modules, CPRN captures the visual details of position reasoning so that the model can achieve more accurate segmentation. To our knowledge, this is the first work that explicitly focuses on position reasoning modeling. We also validate the proposed method on three evaluation datasets. It consistently outperforms existing state-of-the-art methods.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper addresses the localization problem in Referring Image Segmentation. Existing methods mainly focus on the interaction learning between vision and language, but predicting directly in pixel-level space leads to inaccurate localization. The paper proposes a new Collaborative Position Reasoning Network (CPRN), which models entity localization more accurately through Row-Column Interaction (RoCo) and Guided Global Interaction (Holi) modules, thus achieving more accurate segmentation. This is the first work that focuses on position reasoning modeling and its superiority is verified on three datasets.

Collaborative Position Reasoning Network for Referring Image Segmentation

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Joint CRF and Locality-Consistent Dictionary Learning for Semantic Segmentation.

Dual-graph Hierarchical Interaction Network for Referring Image Segmentation

Multi-branch Collaborative Learning Network for 3D Visual Grounding

CoReS: Orchestrating the Dance of Reasoning and Segmentation

Language-Conditioned Region Proposal and Retrieval Network for Referring Expression Comprehension

Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation

Rethinking referring relationships from a perspective of mask-level relational reasoning

Relationship-Embedded Representation Learning for Grounding Referring Expressions

Referring Image Segmentation via Cross-Modal Progressive Comprehension

Hierarchical Collaboration for Referring Image Segmentation

A Multi-Scale Language Embedding Network for Proposal-Free Referring Expression Comprehension.

CRNet: Collaborative Refinement Network for Self-Supervised Video Object Segmentation

Mask Grounding for Referring Image Segmentation

Fully Aligned Network for Referring Image Segmentation

Language-Aware Fine-Grained Object Representation for Referring Expression Comprehension.

A Hybrid Framework for Referring Image Segmentation: Dual-Decoder Model with SAM Complementation

Cross-Modal Recurrent Semantic Comprehension for Referring Image Segmentation

Query Reconstruction Network for Referring Expression Image Segmentation

Image Segmentation with Language Referring Expression and Comprehension