Abstract:High-performance Multimodal Large Language Models (MLLMs) rely heavily on data quality. This study introduces a novel dataset named Img-Diff, designed to enhance fine-grained image recognition in MLLMs by leveraging insights from contrastive learning and image difference captioning. By analyzing object differences between similar images, we challenge models to identify both matching and distinct components. We utilize the Stable-Diffusion-XL model and advanced image editing techniques to create pairs of similar images that highlight object replacements. Our methodology includes a Difference Area Generator for object differences identifying, followed by a Difference Captions Generator for detailed difference descriptions. The result is a relatively small but high-quality dataset of "object replacement" samples. We use the the proposed dataset to finetune state-of-the-art (SOTA) MLLMs such as MGM-7B, yielding comprehensive improvements of performance scores over SOTA models that trained with larger-scale datasets, in numerous image difference and Visual Question Answering tasks. For instance, our trained models notably surpass the SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate alternative methods for generating image difference data through "object removal" and conduct a thorough evaluation to confirm the dataset's diversity, quality, and robustness, presenting several insights on the synthesis of such a contrastive dataset. To encourage further research and advance the field of multimodal data synthesis and enhancement of MLLMs' fundamental capabilities for image understanding, we release our codes and dataset at <a class="link-external link-https" href="https://github.com/modelscope/data-juicer/tree/ImgDiff" rel="external noopener nofollow">this https URL</a>.

15M Multimodal Facial Image-Text Dataset

Face-MLLM: A Large Face Perception Model

Multi-caption Text-to-Face Synthesis: Dataset and Algorithm

Improving Multimodal Datasets with Image Captioning

Face-Cap: Image Captioning using Facial Expression Analysis

OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

Toward High Quality Facial Representation Learning

A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

OmniCorpus: an Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

Image Captioning using Facial Expression and Attention

TextCaps: a Dataset for Image Captioning with Reading Comprehension

Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers

Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

SkinCAP: A Multi-modal Dermatology Dataset Annotated with Rich Medical Captions

From Pixels to Prose: A Large Dataset of Dense Image Captions

LAION-5B: An open large-scale dataset for training next generation image-text models

CapsFusion: Rethinking Image-Text Data at Scale

MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

Image Captioning with Multi-Context Synthetic Data

Descriptive Caption Enhancement with Visual Specialists for Multimodal Perception

CelebV-HQ: A Large-Scale Video Facial Attributes Dataset