Abstract:Cross-modal fusion aims to establish a consistent correspondence between arbitrary modalities. Due to the inherent differences between these modalities, accurately modeling their correspondence is a challenging task. Referring image segmentation (RIS) is a fundamental cross-modal task that intends to segment a desired object from an image based on a given natural language expression. In this paper, we propose an efficient algorithm called the Deep Cross-Modal Fusion Network (DCMFNet) to address this challenge. The proposed algorithm leverages the contextual information from linguistic context to guide the modeling of the visual context, gradually highlighting the referent in the image. The network architecture employs an innovative fusion strategy known as Iterative Gated Fusion (IGF) to capture the consistency relationship between multi-modal features. IGF iteratively adjusts the relative importance of features at each level based on high-level semantics, emphasizing the shared information while suppressing the irrelevant parts. Specifically, IGF consists of cascaded fusion units and gating units. The fusion units integrate high-level semantics with the features from the previous layer to enhance the representation. The gating units perceive the discrepancy between the enhanced features and the original representation, and selectively weight and integrate the important features for further refinement. Through multi-layer iterative optimization, IGF gradually establishes a fine-grained correspondence between arbitrary modalities. Extensive experimental results on the Referring Image Segmentation task demonstrate the effectiveness and utility of the proposed method.

Cross-modal retrieval based on fusion lightweight network.

Iterative graph attention memory network for cross-modal retrieval

DCMFNet: Deep Cross-Modal Fusion Network for Different Modalities with Iterative Gated Fusion

Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation

Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Modality-specific Adaptive Scaling and Attention Network for Cross-Modal Retrieval

Feature Fusion Based on Transformer for Cross-modal Retrieval

Cross‐modal Semantic Correlation Learning by Bi‐CNN Network

Semi-supervised Cross-Modal Learning for Cross Modal Retrieval and Image Annotation.

Deep Memory Network for Cross-Modal Retrieval

HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Semantic Preservation and Hash Fusion Network for Unsupervised Cross-Modal Retrieval

Semantic Consistent Adversarial Cross-Modal Retrieval Exploiting Semantic Similarity

Semantic Guidance Fusion Network for Cross-Modal Semantic Segmentation

Bidirectional Focused Semantic Alignment Attention Network for Cross-Modal Retrieval

Multi-level Similarity Complementary Fusion for Unsupervised Cross-Modal Hashing

Cross-modal Retrieval Via Memory Network.

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Multimedia Feature Mapping and Correlation Learning for Cross-Modal Retrieval

Cross-Modal Hash Method Based on Multi-Scale Fusion and Projection Matching Constraint