Abstract:Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at <a class="link-external link-https" href="https://github.com/Hao840/ADEM-VL" rel="external noopener nofollow">this https URL</a>.

TraVL: Transferring Pre-trained Visual-Linguistic Models for Cross-Lingual Image Captioning.

VLDeformer: Vision–Language Decomposed Transformer for Fast Cross-Modal Retrieval

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

TEVL: Trilinear Encoder for Video-language Representation Learning

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Unified Vision-Language Pre-Training for Image Captioning and VQA

VOLTER: Visual Collaboration and Dual-Stream Fusion for Scene Text Recognition

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective

Generalizing Multimodal Pre-training into Multilingual via Language Acquisition

VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending

Bridging Vision and Language Spaces with Assignment Prediction

Pre-training Model Based on Parallel Cross-Modality Fusion Layer

Visually-Augmented Language Modeling

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features