Abstract:Referring Expression Segmentation (RES), which is aimed at localizing and segmenting the target according to the given language expression, has drawn increasing attention. Existing methods jointly consider the localization and segmentation steps, which rely on the fused visual and linguistic features for both steps. We argue that the conflict between the purpose of identifying an object and generating a mask limits the RES performance. To solve this problem, we propose a parallel position-kernel-segmentation pipeline to better isolate and then interact the localization and segmentation steps. In our pipeline, linguistic information will not directly contaminate the visual feature for segmentation. Specifically, the localization step localizes the target object in the image based on the referring expression, and then the visual kernel obtained from the localization step guides the segmentation step. This pipeline also enables us to train RES in a weakly-supervised way, where the pixel-level segmentation labels are replaced by click annotations on center and corner points. The position head is fully-supervised and trained with the click annotations as supervision, and the segmentation head is trained with weakly-supervised segmentation losses. To validate our framework on a weakly-supervised setting, we annotated three RES benchmark datasets (RefCOCO, RefCOCO+ and RefCOCOg) with click annotations. Our method is simple but surprisingly effective, outperforming all previous state-of-the-art RES methods on fully- and weakly-supervised settings by a large margin. The benchmark code and datasets will be released.

Advancing Referring Expression Segmentation Beyond Single Image

GRES: Generalized Referring Expression Segmentation

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

3D-GRES: Generalized 3D Referring Expression Segmentation

Cascade Grouped Attention Network for Referring Expression Segmentation

Learning to Segment Every Referring Object Point by Point

GSVA: Generalized Segmentation via Multimodal Large Language Models

Towards Omni-supervised Referring Expression Segmentation

Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning

RRSIS: Referring Remote Sensing Image Segmentation

Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherence

RESMatch: Referring Expression Segmentation in a Semi-Supervised Manner

Instance-Aware Generalized Referring Expression Segmentation

Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation

Towards Further Comprehension on Referring Expression with Rationale

Towards Robust Referring Image Segmentation.

Towards Unsupervised Referring Expression Comprehension with Visual Semantic Parsing

RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

Beyond One-to-One: Rethinking the Referring Image Segmentation

Toward Robust Referring Image Segmentation

Meta Compositional Referring Expression Segmentation