Abstract:Recently, multimodal large language models (MM-LLMs) have achieved great success in many multimodal tasks, but their high computational costs limit their further promotion and application. In the MM-LLMs framework, the main computational consumption step is the processing of concatenated text and visual tokens at the LLM layer. The length of the input token for LLM directly affects the overall training and inference efficiency. In response to this issue, we further studied the visual tokens of MM-LLMs. We found that the similarity between visual and CLS tokens in the visual encoder follows a long-tail distribution. In other words, only a few visual tokens are highly similar to CLS tokens. Therefore, we designed a dynamic pruning algorithm to address this issue. Firstly, for different input samples, we search for the inflection point of their visual CLS token similarity curve and use it as the corresponding segmentation point to trim the visual markers. This process mainly reduces the output of the visual encoder to accelerate the model. Then, in the LLM layer, the concatenated visual text tokens are pruned for the second time. During this process, due to the interaction between visual and textual features, visual and textual tokens with low text correlation are further filtered, achieving a balance between efficiency and performance. The results on multiple datasets show that our proposed method can achieve performance that competes with the original performance when using an average of 22% of the original token quantity. Our source code will be made publicly available following acceptance.

Voice-based Local Search Using a Language Model Look-ahead Structure for Efficient Pruning

Reducing time-synchronous beam search effort using stage based look-ahead and language model rank based pruning

AVATAR: Robust Voice Search Engine Leveraging Autoregressive Document Retrieval and Contrastive Learning

Query Expansion Based High Performance Chinese Voice Retrieval

An Efficient Approach of Language Model Applying in ASR Systems

Location-aware Instant Search.

Improving Speech Recognition Accuracy of Local POI Using Geographical Models

SparseLLM: Towards Global Pruning for Pre-trained Language Models

Structured Optimal Brain Pruning for Large Language Models

Personalized language modeling by crowd sourcing with social network data for voice access of cloud applications

FoPru: Focal Pruning for Efficient Large Vision-Language Models

Language Model Estimation For Optimizing End-To-End Performance Of A Natural Language Call Routing System

Balancing Performance and Efficiency: A Multimodal Large Language Model Pruning Method based Image Text Interaction

Linguistic Search Optimization for Deep Learning Based LVCSR

Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models

Large Search Model: Redefining Search Stack in the Era of LLMs

Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model

Massive End-to-end Models for Short Search Queries

Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search

Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency

Mind Your Language: Effects of Spoken Query Formulation on Retrieval Effectiveness