Abstract:High-performance Multimodal Large Language Models (MLLMs) rely heavily on data quality. This study introduces a novel dataset named Img-Diff, designed to enhance fine-grained image recognition in MLLMs by leveraging insights from contrastive learning and image difference captioning. By analyzing object differences between similar images, we challenge models to identify both matching and distinct components. We utilize the Stable-Diffusion-XL model and advanced image editing techniques to create pairs of similar images that highlight object replacements. Our methodology includes a Difference Area Generator for object differences identifying, followed by a Difference Captions Generator for detailed difference descriptions. The result is a relatively small but high-quality dataset of "object replacement" samples. We use the the proposed dataset to finetune state-of-the-art (SOTA) MLLMs such as MGM-7B, yielding comprehensive improvements of performance scores over SOTA models that trained with larger-scale datasets, in numerous image difference and Visual Question Answering tasks. For instance, our trained models notably surpass the SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate alternative methods for generating image difference data through "object removal" and conduct a thorough evaluation to confirm the dataset's diversity, quality, and robustness, presenting several insights on the synthesis of such a contrastive dataset. To encourage further research and advance the field of multimodal data synthesis and enhancement of MLLMs' fundamental capabilities for image understanding, we release our codes and dataset at <a class="link-external link-https" href="https://github.com/modelscope/data-juicer/tree/ImgDiff" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main goal of this paper is to propose a new data synthesis method and corresponding dataset to enhance the performance of Multimodal Large Language Models (MLLMs) in fine-grained image recognition tasks. Specifically, the research team developed a dataset called Img-Diff, which contains a series of highly similar but subtly different image pairs, accompanied by text describing the differences between these images. The key contributions of the paper are as follows: 1. **Proposed a new data synthesis method**: By using advanced image editing techniques and generative models to create image pairs with small object replacements, along with detailed text descriptions of these differences. 2. **Created the Img-Diff dataset**: This dataset consists of a series of highly similar but subtly different image pairs, aimed at training models to recognize these small differences. The construction process of this dataset includes multiple stages, such as image segmentation, filtering, and the generation of detailed image difference descriptions. 3. **Evaluated the improvement in model performance**: Researchers fine-tuned existing SOTA MLLMs (e.g., LLaVA-1.5 and MGM) using the Img-Diff dataset and evaluated the performance of these fine-tuned models on multiple benchmarks. The results showed significant improvements in tasks such as MMVP, Spot-the-Diff, and Image-Edit-Request. 4. **Comprehensive dataset evaluation**: The paper also conducted a thorough evaluation of the quality and diversity of the Img-Diff dataset, ensuring it covers a wide range of object categories and exhibits rich variability. 5. **Open-source code and dataset**: To facilitate further research and development, the authors have made their code and dataset publicly available. In summary, by introducing the Img-Diff dataset, this paper opens up a new avenue for enhancing the performance of MLLMs in image understanding tasks, particularly in fine-grained image recognition and image difference recognition.

Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations

Multimodal Data Augmentation for Image Captioning using Diffusion Models

Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining

OneDiff: A Generalist Model for Image Difference Captioning

Describing Differences in Image Sets with Natural Language

DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training

CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset

CSDNet: Contrastive Similarity Distillation Network for Multi-lingual Image-Text Retrieval.

UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning

WuDaoMM: A large-scale Multi-Modal Dataset for Pre-training models

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation

15M Multimodal Facial Image-Text Dataset

Towards Open-ended Visual Quality Comparison

MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration