Abstract:The image-text retrieval task aims to retrieve relevant information from a given image or text. The main challenge is to unify multimodal representation and distinguish fine-grained differences across modalities, thereby finding similar contents and filtering irrelevant contents. However, existing methods mainly focus on unified semantic representation and concept alignment for multi-modalities, while the fine-grained differences across modalities have rarely been studied before, making it difficult to solve the information asymmetry problem. In this paper, we propose a novel asymmetry-sensitive contrastive learning method. By generating corresponding positive and negative samples for different asymmetry types, our method can simultaneously ensure fine-grained semantic differentiation and unified semantic representation between multi-modalities. Additionally, a hierarchical cross-modal fusion method is proposed, which integrates global and local-level features through a multimodal attention mechanism to achieve concept alignment. Extensive experiments performed on MSCOCO and Flickr30K, demonstrate the effectiveness and superiority of our proposed method.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to address the issue of information asymmetry in image-text retrieval tasks. Specifically, existing methods mainly focus on the unification of multimodal semantic representations and concept alignment, but rarely study the fine-grained differences between different modalities. This information asymmetry problem leads to a situation where one modality may contain more or less information than the other when describing the same scene. For example, the text may contain redundant information not present in the image, or the image may contain details not mentioned in the text. To tackle this challenge, the authors propose a novel Asymmetry-Sensitive Contrastive Learning method (AsCL), which ensures fine-grained semantic distinction and unification of multimodal semantic representations by generating different types of information asymmetry samples. Additionally, a hierarchical cross-modal fusion method is proposed, which integrates global and local features through a multimodal attention mechanism to achieve concept alignment. ### Main Contributions: 1. **Asymmetry-Sensitive Contrastive Learning Method**: Generates corresponding positive and negative samples for each type of fine-grained information asymmetry to achieve better cross-modal retrieval based on semantic similarity. 2. **Image-Text Feature Fusion and Semantic Alignment Method**: Achieves high-quality modality interaction from both global and local perspectives based on a cross-modal attention mechanism. 3. **Experimental Validation**: Extensive experiments demonstrate that the proposed method outperforms existing methods on multiple benchmark datasets. ### Types of Information Asymmetry: - **Asymmetry Type I**: The text contains redundant information not present in the image. - **Asymmetry Type II**: The text contains more relevant information corresponding to the image. - **Asymmetry Type III**: The text omits some relevant information corresponding to the image. Through these methods, the paper effectively addresses the information asymmetry problem in multimodal retrieval tasks, improving the accuracy and robustness of retrieval.

AsCL: An Asymmetry-sensitive Contrastive Learning Method for Image-Text Retrieval with Cross-Modal Fusion

Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval

Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training

On the Generalization of Multi-modal Contrastive Learning

CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

Iterative Uni-modal and Cross-modal Clustered Contrastive Learning for Image-text Retrieval

Bridging the gap: dual perception attention and local-global similarity fusion for cross-modal image-text matching

Image–Text Cross-Modal Retrieval with Instance Contrastive Embedding

Cross-modal contrastive learning for multimodal sentiment recognition

Cross-Modal Attention With Semantic Consistence for Image–Text Matching

Masking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Cross-Modal Attention Preservation with Self-Contrastive Learning for Composed Query-Based Image Retrieval

Cross-modal Image Retrieval with Deep Mutual Information Maximization

Cross-Modal Image-Text Retrieval with Semantic Consistency

Text-Centric Multimodal Contrastive Learning for Sentiment Analysis

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Contrastive Latent Space Reconstruction Learning for Audio-Text Retrieval

Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment

Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval