Abstract:Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at <a class="link-external link-https" href="https://github.com/Hao840/ADEM-VL" rel="external noopener nofollow">this https URL</a>.

A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach

XlanV Model with Adaptively Multi-Modality Feature Fusing for Video Captioning

VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Unified Vision-Language Pre-Training for Image Captioning and VQA

Pre-training Model Based on Parallel Cross-Modality Fusion Layer

An Empirical Study of Training End-to-End Vision-and-Language Transformers

Bridging the Gap Between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA

Modular dual-stream visual fusion network for visual question answering

Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

TraVL: Transferring Pre-trained Visual-Linguistic Models for Cross-Lingual Image Captioning.

Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA

Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering