Abstract:In this work, we design a Multi‐scale Gradient balanced Central Difference Convolution (MG‐CDC) and a Graph convolutional network‐based Language and Image Fusion (GLIF) for cross‐modal encoder. Since our encoder achieves robust cross‐modal alignment and context mining, we could use a light‐weight decoder for segmentation prediction. Extensive experiments show that our method outperforms the state‐of‐the‐art methods on three public datasets. Referring image segmentation identifies the object masks from images with the guidance of input natural language expressions. Nowadays, many remarkable cross‐modal decoder are devoted to this task. But there are mainly two key challenges in these models. One is that these models usually lack to extract fine‐grained boundary information and gradient information of images. The other is that these models usually lack to explore language associations among image pixels. In this work, a Multi‐scale Gradient balanced Central Difference Convolution (MG‐CDC) and a Graph convolutional network‐based Language and Image Fusion (GLIF) for cross‐modal encoder, called Graph‐RefSeg, are designed. Specifically, in the shallow layer of the encoder, the MG‐CDC captures comprehensive fine‐grained image features. It could enhance the perception of target boundaries and provide effective guidance for deeper encoding layers. In each encoder layer, the GLIF is used for cross‐modal fusion. It could explore the correlation of every pixel and its corresponding language vectors by a graph neural network. Since the encoder achieves robust cross‐modal alignment and context mining, a light‐weight decoder could be used for segmentation prediction. Extensive experiments show that the proposed Graph‐RefSeg outperforms the state‐of‐the‐art methods on three public datasets. Code and models will be made publicly available at https://github.com/ZYQ111/Graph_refseg.

DCMFNet: Deep Cross-Modal Fusion Network for Referring Image Segmentation with Iterative Gated Fusion

DCMFNet: Deep Cross-Modal Fusion Network for Different Modalities with Iterative Gated Fusion

CMF: Cascaded Multi-Model Fusion for Referring Image Segmentation

Multiscale Deep Feature Selection Fusion Network for Referring Image Segmentation

Structured Multimodal Fusion Network for Referring Image Segmentation

Cross-Modal Fusing Vision-Language Network for Referring Image Segmentation

Referring Image Segmentation with Fine-Grained Semantic Funneling Infusion

CIMFNet: Cross-layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Global Context Enhanced Multi-modal Fusion for Referring Image Segmentation

CMIRNet: Cross-Modal Interactive Reasoning Network for Referring Image Segmentation

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

CIGF-Net: Cross-Modality Interaction and Global-Feature Fusion for RGB-T Semantic Segmentation

MCFNet: Multi-scale Covariance Feature Fusion Network for Real-time Semantic Segmentation

Cross‐modal fusion encoder via graph neural network for referring image segmentation

Correlation-Guided Discriminative Cross-Modality Features Network for Infrared and Visible Image Fusion

A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

DCFusion: Dual-Headed Fusion Strategy and Contextual Information Awareness for Infrared and Visible Remote Sensing Image

FDGSNet: A Multi-modal Gated Segmentation Network for Remote Sensing Image Based on Frequency Decomposition

Multiscale Fusion Convolutional Network in Real-time Semantic Segmentation

Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data