Abstract:Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at <a class="link-external link-https" href="https://github.com/Hao840/ADEM-VL" rel="external noopener nofollow">this https URL</a>.

KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

Benchmarking VLMs' Reasoning About Persuasive Atypical Images

Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models

Adapting Vision-Language Models via Learning to Inject Knowledge

Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models

Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment

AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation

Automatic Understanding of Image and Video Advertisements

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation

Visually-Augmented Language Modeling

Disease-informed Adaptation of Vision-Language Models

Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding

Few-shot Adaptation of Medical Vision-Language Models

Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions

Unsupervised Domain Adaption Harnessing Vision-Language Pre-training

Rethinking Image-to-Video Adaptation: An Object-centric Perspective

Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities