Abstract:Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model. LMMs typically take in a fixed and large amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly. However, due to the inherent design of the Transformer architecture, the computational costs of these models tend to increase quadratically with the number of input tokens. To tackle this problem, we explore a token reduction mechanism that identifies significant spatial redundancy among visual tokens. In response, we propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs. Specifically, to metric the importance of each token, we exploit the sparsity observed in the visual encoder, characterized by the sparse distribution of attention scores between the class token and visual tokens. This sparsity enables us to dynamically select the most crucial visual tokens to retain. Subsequently, we cluster the selected (unpruned) tokens based on their key similarity and merge them with the unpruned tokens, effectively supplementing and enhancing their informational content. Empirically, when applied to LLaVA-1.5, our approach can compress the visual tokens by 14 times on average, and achieve comparable performance across diverse visual question-answering and reasoning tasks. Code and checkpoints are at

What problem does this paper attempt to address?

The paper primarily addresses the issue of computational efficiency in large multimodal models (LMMs). Specifically, it focuses on how to reduce the number of visual tokens input into the language model without sacrificing model performance, thereby lowering computational costs. Current LMMs exhibit strong visual reasoning capabilities by connecting visual encoders with large language models. These models typically receive a fixed and large number of visual tokens as input, such as features from the penultimate layer of the CLIP visual encoder. As the models handle more complex visual inputs, such as high-resolution images and videos, the number of visual tokens increases significantly, leading to a quadratic growth in computational costs for Transformer-based models. To address this issue, the paper proposes the PruMerge method, a novel adaptive visual token reduction strategy. PruMerge can significantly reduce the number of visual tokens while maintaining the performance of LMMs. The specific approach includes: 1. **Important Token Selection**: Utilizing the sparsity of attention scores in the visual encoder to measure the importance of each token and dynamically selecting key visual tokens to retain. 2. **Token Merging**: Clustering the selected (unpruned) tokens and merging them with the unpruned tokens to supplement and enhance their informational content. Experimental results show that when applied to the LLaVA-1.5 model, PruMerge can compress visual tokens by an average of approximately 14 times while maintaining comparable performance across various visual question answering and reasoning tasks. Additionally, PruMerge demonstrates its generality and efficiency across different modalities of data, particularly in video understanding tasks, where integrating PruMerge only during the inference stage can accelerate the processing speed of video LMMs and improve their performance. In summary, this research aims to improve the overall efficiency of LMMs by reducing the number of visual tokens without affecting the model's reasoning capabilities.

LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See

Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction

AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning

iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

Efficient Multi-modal Large Language Models via Visual Token Grouping

Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings

TokenPacker: Efficient Visual Projector for Multimodal LLM

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models

Balancing Performance and Efficiency: A Multimodal Large Language Model Pruning Method based Image Text Interaction

freePruner: A Training-free Approach for Large Multimodal Model Acceleration

FoPru: Focal Pruning for Efficient Large Vision-Language Models

Efficient Large Multi-modal Models via Visual Context Compression

ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models

[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster

[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs

Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving

Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference

Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy