Abstract:Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at <a class="link-external link-https" href="https://github.com/Hao840/ADEM-VL" rel="external noopener nofollow">this https URL</a>.

Fine-Grained Visual Prompt Learning of Vision-Language Models for Image Recognition

Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting

Mutual Prompt Leaning for Vision Language Models

Towards Better Vision-Inspired Vision-Language Models

Promoting AI Equity in Science: Generalized Domain Prompt Learning for Accessible VLM Research

Concept-Guided Prompt Learning for Generalization in Vision-Language Models

LaViP:Language-Grounded Visual Prompts

DPL: Decoupled Prompt Learning for Vision-Language Models

Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment

Towards Generative Class Prompt Learning for Few-shot Visual Recognition

Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning

FineFMPL: Fine-grained Feature Mining Prompt Learning for Few-Shot Class Incremental Learning

In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model

Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge

Revisiting Prompt Pretraining of Vision-Language Models

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Multi-modal Attribute Prompting for Vision-Language Models

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

LOBG:Less Overfitting for Better Generalization in Vision-Language Model