Abstract:Existing Multimodal Large Language Models (MLLMs) increasingly emphasize complex understanding of various visual elements, including multiple objects, text information, and spatial relations. Their development for comprehensive visual perception hinges on the availability of high-quality image-text datasets that offer diverse visual elements and throughout image descriptions. However, the scarcity of such hyper-detailed datasets currently hinders progress within the MLLM community. The bottleneck stems from the limited perceptual capabilities of current caption engines, which fall short in providing complete and accurate annotations. To facilitate the cutting-edge research of MLLMs on comprehensive vision perception, we thereby propose Perceptual Fusion, using a low-budget but highly effective caption engine for complete and accurate image descriptions. Specifically, Perceptual Fusion integrates diverse perception experts as image priors to provide explicit information on visual elements and adopts an efficient MLLM as a centric pivot to mimic advanced MLLMs' perception abilities. We carefully select 1M highly representative images from uncurated LAION dataset and generate dense descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments validate that our engine outperforms its counterparts, where the resulting dataset significantly improves the perception and cognition abilities of existing MLLMs across diverse vision-language benchmarks, especially with high-resolution images as inputs. The dataset and code are publicly available at <a class="link-external link-https" href="https://github.com/baaivision/DenseFusion" rel="external noopener nofollow">this https URL</a>.

Data-Efficient Multimodal Fusion on a Single GPU

Attention Bottlenecks for Multimodal Fusion

Deep Multimodal Data Fusion

GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer

Adaptive Fusion Techniques for Multimodal Data

MixFuse: an Iterative Mix-Attention Transformer for Multi-Modal Image Fusion

Enhancing Perception Capabilities of Multimodal LLMs with Training-free Fusion

Cyclic Autoencoder for Multimodal Data Alignment Using Custom Datasets

Multimodal Fusion Method Based on Self-Attention Mechanism

Equivariant Multi-Modality Image Fusion

MMTM: Multimodal Transfer Module for CNN Fusion

Generalized Multimodal Fusion via Poisson-Nernst-Planck Equation

Multimodal Token Fusion for Vision Transformers

DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

CMFuse: Cross-Modal Features Mixing Via Convolution and MLP for Infrared and Visible Image Fusion

Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

From Unimodal to Multimodal: Scaling up Projectors to Align Modalities