Abstract:Current multimodal models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives permute terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across a wide range of vision-language datasets, including our InpaintCOCO dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that current vision - language models have difficulty in developing fine - grained conceptual understanding in contrastive learning. Specifically, due to the use of random negative samples in the pre - training process, the loss function almost only compares very dissimilar concepts, resulting in poor performance of the model when dealing with fine - grained semantic differences. To improve this situation, the author proposes a new pre - training method, which enhances the model's fine - grained alignment ability of visual and textual concepts by introducing synthetic hard negative samples. In addition, the author also creates a new dataset named InpaintCOCO, which is used to evaluate the fine - grained alignment performance of vision - language models in terms of color, object, and size, etc. ### Main contributions of the paper: 1. **Propose a new contrastive learning method**: By using hard negative samples in the contrastive learning objective, the model can focus on refining its understanding of concepts. 2. **By introducing hard negative samples in the language part**: Force the model to learn the correct visual and language alignment. This method can improve multi - modal performance only by operating on the language side. 3. **Create a challenging dataset**: Evaluate the model's conceptual understanding ability from a visual perspective by generating adversarial samples. This dataset contains more than 1,260 adversarial samples, which change the visual concepts by generative inpainting of COCO images so that they no longer match the original image descriptions. ### Method details: - **Generation of hard negative samples**: Generate hard negative samples by replacing keywords in image - text pairs, breaking the alignment relationship between image content and description. For example, replace "white cat" with "brown cat", or "cat" with "dog". - **Training details**: Use CLIP's image and text encoders (ViT - B/32 model), with a batch size of 64, Adam optimizer, a weight decay of 0.1, and a learning rate of 5×10⁻⁶. The training time is less than two hours, and the GPU memory consumption is less than 12GB. - **Dataset**: Continue to use CLIP for pre - training on the COCO image caption dataset, and filter out samples containing specific keywords for keyword replacement. ### Experimental results: - **Fine - grained conceptual understanding**: In concepts such as object, color, position, and size, the method using hard negative samples significantly improves the model's fine - grained conceptual understanding ability. - **General image retrieval ability**: Although hard negative samples are introduced, the model's general image retrieval ability hardly decreases. - **Evaluation on the InpaintCOCO dataset**: Further verifies the model's fine - grained conceptual understanding ability from a visual perspective, especially in terms of object, color, and size. ### Conclusion: By introducing hard negative samples, this method can significantly improve the fine - grained conceptual understanding ability of vision - language models without significantly affecting the general retrieval ability. This method has shown superior performance on multiple datasets, especially in terms of fine - grained conceptual understanding.

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

SynCo: Synthetic Hard Negatives in Contrastive Learning for Better Unsupervised Visual Representations

Synthetic Hard Negative Samples for Contrastive Learning

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

ITContrast: contrastive learning with hard negative synthesis for image-text matching

Multimodal Contrastive Training for Visual Representation Learning

When hard negative sampling meets supervised contrastive learning

Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining

Contrastive Learning of Visual-Semantic Embeddings

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding

AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries

Robust image representations with counterfactual contrastive learning

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

HAPiCLR: heuristic attention pixel-level contrastive loss representation learning for self-supervised pretraining

TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

Adaptive Multi-head Contrastive Learning

Rethinking Positive Pairs in Contrastive Learning

The Hard Positive Truth about Vision-Language Compositionality

Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation

On the Importance of Contrastive Loss in Multimodal Learning