Abstract:Referring segmentation is a fundamental vision-language task that aims to segment out an object from an image or video in accordance with a natural language description. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image or video frames. A paradigm for tackling this problem in both the image and the video domains is to leverage a powerful vision-language ("cross-modal") decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advances in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. Based on the idea of conducting cross-modal feature fusion in the visual feature encoding stage, we propose a unified framework named Language-Aware Vision Transformer (LAVT), which leverages the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results can be harvested with a light-weight mask predictor. One of the key components in the proposed system is a dense attention mechanism for collecting pixel-specific linguistic cues. When dealing with video inputs, we present the video LAVT framework and design a 3D version of this component by introducing multi-scale convolutional operators arranged in a parallel fashion, which can exploit spatio-temporal dependencies at different granularity levels. We further introduce unified LAVT as a unified framework capable of handling both image and video inputs, with enhanced segmentation capabilities for the unified referring segmentation task. Our methods surpass previous state-of-the-art methods on seven benchmarks for referring image segmentation and referring video segmentation. The code to reproduce our experiments is available at LAVT-RS.

Language Query Based Transformer with Multi-Scale Cross-Modal Alignment for Visual Grounding on Remote Sensing Images

Language Query-Based Transformer With Multiscale Cross-Modal Alignment for Visual Grounding on Remote Sensing Images

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

Visual Selection and Multistage Reasoning for RSVG

Language-Guided Progressive Attention for Visual Grounding in Remote Sensing Images

Injecting Linguistic into Visual Backbone: Query-Aware Multimodal Fusion Network for Remote Sensing Visual Grounding

Transformer-based Visual Grounding with Cross-modality Interaction

Cross-modal Transformer with Language Query for Referring Image Segmentation

A Spatial Frequency Fusion Strategy Based on Linguistic Query Refinement for RSVG

Rethinking Transformers for Semantic Segmentation of Remote Sensing Images.

Attribute-Prompting Multi-Modal Object Reasoning Transformer for Remote Sensing Visual Grounding

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Alignment and Multimodal Reasoning for Remote Sensing Visual Question Answering

Language Conditioned Multi-Scale Visual Attention Networks for Visual Grounding

Language-Aware Vision Transformer for Referring Segmentation

SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion

Transcending Fusion: A Multiscale Alignment Method for Remote Sensing Image–Text Retrieval

Unified Transformer with Cross-Modal Mixture Experts for Remote-Sensing Visual Question Answering

A Multiscale Grouping Transformer With CLIP Latents for Remote Sensing Image Captioning

Transcending Fusion: A Multi-Scale Alignment Method for Remote Sensing Image-Text Retrieval