Abstract:Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the same object, leading to incomplete masks and localization errors in predictions. To tackle this problem, we propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word-pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of linguistic and pixel-level features in intermediate layers of the vision and language encoders. Based on the word-pixel aligned embedding, a set of mask proposals are generated to hypothesize possible objects. Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e.g., RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of-the-art methods, e.g., about 2% oIoU increase on the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class.

Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment

Cross-Modal Semantic Alignment before Fusion for Two-Pass End-to-End Spoken Language Understanding

Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling Via Adjustive and Forced Cross-Task Alignment

Area-keywords Cross-Modal Alignment for Referring Image Segmentation

Unified Lexical Representation for Interpretable Visual-Language Alignment

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

SLAN: Self-Locator Aided Network for Cross-Modal Understanding

Word Alignment by Fine-tuning Embeddings on Parallel Corpora

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP

Fine-Granularity Alignment for Text-Based Person Retrieval Via Semantics-Centric Visual Division

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

CLIM: Contrastive Language-Image Mosaic for Region Representation

COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment

Text-centric Alignment for Multi-Modality Learning

WAD-X: Improving Zero-shot Cross-lingual Transfer via Adapter-based Word Alignment

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation