Abstract:Referring expression comprehension aims to align natural language queries with visual scenes, which requires establishing fine-grained correspondence between vision and language. This has important applications in multi-modal reasoning systems. Existing methods typically use text-agnostic visual backbones to extract features independently without considering the specific text input. However, we argue that the extracted visual features can be inconsistent with the referring expression, which hurts multi-modal understanding. To address this, we first propose Query-modulated Refinement Network (QRNet) that leverages language guidance to guide visual feature extraction. However, it only focuses on the grounding task that can only provide coarse-grained annotations in the form of bounding box coordinates. The guidance for the visual backbone is indirect, and the inconsistent issue still exists. To this end, we further propose UniQRNet, a multi-task framework over the QRNet to learn referring expression grounding and segmentation jointly. The framework introduces a multi-task head that leverages fine-grained pixel-level supervision from the segmentation task to directly guide the intermediate layers of QRNet to learn text-consistent visual features. Besides, UniQRNet also includes a loss balance strategy that allows two types of supervision signals to cooperate and optimize the model together. We conduct the most comprehensive comparison experiment covering four major datasets, ten evaluation set and three evaluation metrics used in previous work. UniQRNet outperforms previous state-of-the-art methods by a large margin on both referring comprehensive grounding (1.8%~5.09%) and segmentation tasks (0.57%~5.56%). Ablation and analysis reveal that UniQRNet can improve the consistency of visual features with text input and can bring significant performance improvement.

Unified Referring Expression Generation for Bounding Boxes and Segmentations

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

A proposal-free one-stage framework for referring expression comprehension and generation via dense cross-attention

A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation

Whether you can locate or not? Interactive Referring Expression Generation

A Direct Regression Scene Text Detector with Position-Sensitive Segmentation

Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding

Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding

Transferrable Referring Expression Grounding with Concept Transfer and Context Inheritance

UniQRNet: Unifying Referring Expression Grounding and Segmentation with QRNet

Multi-level expression guided attention network for referring expression comprehension

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning

Resolving Referring Expressions in Images With Labeled Elements

Image Segmentation with Language Referring Expression and Comprehension

Key-Word-Aware Network for Referring Expression Image Segmentation

Advancing Referring Expression Segmentation Beyond Single Image

Boundary Knowledge Translation Based Reference Semantic Segmentation

Locate then Segment: A Strong Pipeline for Referring Image Segmentation