DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

Xiaotong Li,Fan Zhang,Haiwen Diao,Yueze Wang,Xinlong Wang,Ling-Yu Duan

2024-07-11

Abstract:Existing Multimodal Large Language Models (MLLMs) increasingly emphasize complex understanding of various visual elements, including multiple objects, text information, and spatial relations. Their development for comprehensive visual perception hinges on the availability of high-quality image-text datasets that offer diverse visual elements and throughout image descriptions. However, the scarcity of such hyper-detailed datasets currently hinders progress within the MLLM community. The bottleneck stems from the limited perceptual capabilities of current caption engines, which fall short in providing complete and accurate annotations. To facilitate the cutting-edge research of MLLMs on comprehensive vision perception, we thereby propose Perceptual Fusion, using a low-budget but highly effective caption engine for complete and accurate image descriptions. Specifically, Perceptual Fusion integrates diverse perception experts as image priors to provide explicit information on visual elements and adopts an efficient MLLM as a centric pivot to mimic advanced MLLMs' perception abilities. We carefully select 1M highly representative images from uncurated LAION dataset and generate dense descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments validate that our engine outperforms its counterparts, where the resulting dataset significantly improves the perception and cognition abilities of existing MLLMs across diverse vision-language benchmarks, especially with high-resolution images as inputs. The dataset and code are publicly available at <a class="link-external link-https" href="https://github.com/baaivision/DenseFusion" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper focuses on how to enhance the comprehensive visual perception capability of Multimodal Large Language Models (MLLMs). Existing MLLMs face challenges in understanding and processing various visual elements, requiring high-quality image-text datasets to provide rich visual descriptions. However, the scarcity of such highly-detailed datasets hinders research progress. The paper proposes a method called "Perceptual Fusion" to address this issue by using a low-cost but efficient caption engine to generate complete and accurate image descriptions, bridging the gap in capturing complex visual information for current caption engines.

Perceptual Fusion combines different visual experts such as object detection, image captioning, and text recognition models as image priors, utilizing a low-budget MLLM as the central point to simulate the perception capability of advanced MLLMs. The paper selects 1 million representative images from the unfiltered LAION dataset and generates dense descriptions using the proposed engine, constructing a dataset called DenseFusion-1M. Experimental results demonstrate that DenseFusion-1M significantly enhances the perception and cognition abilities of existing MLLMs in various visual-language benchmark tests, especially with high-resolution image inputs.

The main contributions of the paper include:
1. Proposing a perceptual fusion pipeline that utilizes multiple expert sources as image priors to create a powerful low-budget caption engine.
2. Constructing a large-scale highly-detailed image-text dataset, DenseFusion-1M, containing rich text information, multiple objects, attributes, spatial relationships, etc.
3. Demonstrating that MLLMs trained based on DenseFusion-1M outperform existing state-of-the-art models in multiple visual-language tasks.

Through this approach, the paper aims to promote a comprehensive understanding of complex visual elements and provide more accurate visual-language alignment.